[Emerging-Sigs] Fake AV sigs.

Paul Halliday paul.halliday at gmail.com
Thu Oct 7 11:55:48 EDT 2010

On Thu, Oct 7, 2010 at 12:25 PM, Martin Holste <mcholste at gmail.com> wrote:

> I recommend logging all URL activity (if possible) and making it
> searchable.  If you're running a web proxy, then you've probably
> already discovered a way to data mine through the backend, be it SQL
> or flat file.  However, I have a 500 Mb/sec pipe to monitor, so I need
> something a little bigger.  My solution is a wrapper around httpry
> which matches requests and response and does some GeoIP tagging as
> well.  Then I syslog-ify that and stream it to our centralized log
> management where we can efficiently query for it (our logs are
> full-text indexed with Sphinx).  Since we also send our Snort alerts
> to syslog, a query for an IP will bring up the Snort alert right next
> to the URL when sorted by time.  Httpry is incredibly fast, so we're
> logging about 3300 URL's/second on old hardware and only using about
> 40% of one CPU.
> I'd be happy to share my httpry logger script if with anyone--shoot me
> an email off-list.  I suppose it could be trivially modified to log
> directly to file or SQL.  I also plan on putting my syslog indexing
> cluster app up on Sourceforge eventually.
I have already been playing with this as well. I use urlsnarf from the
dsniff package on a 200MB link. I put the results into SQL with a little
parser I made here:


The problem is, I am struggling to get it all together and quickly usable.
It is a hell of a lot of data. I have been looking at Sphinx; in fact as
recently as yesterday to try an put what I have together.

I would be interested in any Sphinx info that you would be willing to share.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.emergingthreats.net/pipermail/emerging-sigs/attachments/20101007/2ecf1910/attachment.html

More information about the Emerging-sigs mailing list