[Emerging-Sigs] Fake AV sigs.
paul.halliday at gmail.com
Thu Oct 7 11:55:48 EDT 2010
On Thu, Oct 7, 2010 at 12:25 PM, Martin Holste <mcholste at gmail.com> wrote:
> I recommend logging all URL activity (if possible) and making it
> searchable. If you're running a web proxy, then you've probably
> already discovered a way to data mine through the backend, be it SQL
> or flat file. However, I have a 500 Mb/sec pipe to monitor, so I need
> something a little bigger. My solution is a wrapper around httpry
> which matches requests and response and does some GeoIP tagging as
> well. Then I syslog-ify that and stream it to our centralized log
> management where we can efficiently query for it (our logs are
> full-text indexed with Sphinx). Since we also send our Snort alerts
> to syslog, a query for an IP will bring up the Snort alert right next
> to the URL when sorted by time. Httpry is incredibly fast, so we're
> logging about 3300 URL's/second on old hardware and only using about
> 40% of one CPU.
> I'd be happy to share my httpry logger script if with anyone--shoot me
> an email off-list. I suppose it could be trivially modified to log
> directly to file or SQL. I also plan on putting my syslog indexing
> cluster app up on Sourceforge eventually.
I have already been playing with this as well. I use urlsnarf from the
dsniff package on a 200MB link. I put the results into SQL with a little
parser I made here:
The problem is, I am struggling to get it all together and quickly usable.
It is a hell of a lot of data. I have been looking at Sphinx; in fact as
recently as yesterday to try an put what I have together.
I would be interested in any Sphinx info that you would be willing to share.
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the Emerging-sigs