Author Topic: Need Advice on Cron Job  (Read 18180 times)

0 Members and 2 Guests are viewing this topic.

January 07, 2018, 12:59:04 PM

RazorBurn

  • Newbie

  • Offline
  • *

  • 8
  • Reputation:
    0
    • View Profile
Need Advice on Cron Job
« on: January 07, 2018, 12:59:04 PM »
Ive successfully deployed MRF on WSL in Windows 10 64 bit Fall Creators update..



Problem is with my huge collection around 1TB of malware. and with only public API key for Virustotal, Im limited to 4 queries per minute.. A lot of sample have no VT scan results and had to manualy click the VT scan button with missing VT scan results..

Is their a command to invoke VT scan query for missing results for some samples, or if possible, give me the steps for Cron job, the Docs about cron in the Documentation is not sufficient..

Reply #1January 07, 2018, 01:29:26 PM

RazorBurn

  • Newbie

  • Offline
  • *

  • 8
  • Reputation:
    0
    • View Profile
Re: Need Advice on Cron Job
« Reply #1 on: January 07, 2018, 01:29:26 PM »
Sry, found a way to invoke cron using

Code: [Select]
http://localhost/mrf/cron.php?token=edfe238e15c964e8a8218cf218e43dc1
Problem is its only doing the Cuckoo Analysis



Would it be possible to skip the Cuckoo analysis as I dont have Cuckoo running yet..

Reply #2January 09, 2018, 08:05:01 AM

Tigzy

  • Administrator
  • Hero Member

  • Offline
  • *****

  • 954
  • Reputation:
    90
  • Personal Text
    Owner, Adlice Software
    • View Profile
    • Adlice Software
Re: Need Advice on Cron Job
« Reply #2 on: January 09, 2018, 08:05:01 AM »
Hello,
Can you show me the config file sections for VirusTotal and Cuckoo (please redact your API key !)

Thanks,

Reply #3January 09, 2018, 02:56:03 PM

RazorBurn

  • Newbie

  • Offline
  • *

  • 8
  • Reputation:
    0
    • View Profile
Re: Need Advice on Cron Job
« Reply #3 on: January 09, 2018, 02:56:03 PM »
Code: [Select]
"cuckoo" => array(
"enabled" => True,
"class" => "Cuckoo",
"priority" => 10,
"api_base_url" => 'http://localhost:8090/',
"web_base_url" => 'http://localhost:8000/',
"scan" => array(
//"package" => "",    // uncomment to use
//"timeout" => "",    // uncomment to use
//"priority" => 3,    // 1 to 3, uncomment to use
//"options" => "",    // uncomment to use
//"machine" => "",    // uncomment to use
//"platform" => "",   // uncomment to use
//"tags" => "mrf",       // uncomment to use
//"custom" => "",     // uncomment to use
//"owner" => "",      // uncomment to use
//"memory" => False   // uncomment to use
),
"scan_optional" => array(
//"options" => [ "option1", "option2" ],    // uncomment to use
)
),
"virustotal" => array(
"enabled" => True,
"class" => "VirusTotal",
"priority" => 10,
"key" => 'My Secret Key',
"automatic_upload" => True,
"comment_uploaded" => array(
"enabled" => False, // If true, files uploaded (new analysis) will be commented upon completion
"comment" => "Some comment you want to put in VirusTotal"
),
"vendors_priority" => array( // List of vendors as seen in the VT API, the first one detecting will give its threat name to the sample
"Kaspersky",
"Microsoft",
"BitDefender",
"DrWeb ",
                "Symantec",
                "Ikarus",
"ESET-NOD32"
)
)

A small suggestion, Can the binary storage be Segmented by the First 2 to 3 Chars of SHA1 of MD5 like Viper do? Having 200k to 300k files in one folder is bad for slow HDD.. Also, can you put the Vendor name used for the Threat Name?



Also can someone who studies malware as a hobby can have lower subcription? $50 a month is too much for me.. When all is well and MRF, Cuckoo, and Viper working well together, I'll be deploying this to a Linux 2U rack and Subscibe if the price is reasonable for me..
« Last Edit: January 09, 2018, 03:01:30 PM by RazorBurn »

Reply #4January 09, 2018, 06:09:20 PM

Tigzy

  • Administrator
  • Hero Member

  • Offline
  • *****

  • 954
  • Reputation:
    90
  • Personal Text
    Owner, Adlice Software
    • View Profile
    • Adlice Software
Re: Need Advice on Cron Job
« Reply #4 on: January 09, 2018, 06:09:20 PM »
Ok, if you want to disable Cuckoo:
"enabled" => False,

With that switched off, can you tell me if cron catches any VirusTotal update?

Quote
A small suggestion, Can the binary storage be Segmented by the First 2 to 3 Chars of SHA1 of MD5 like Viper do? Having 200k to 300k files in one folder is bad for slow HDD..
Good idea.

Quote
Also, can you put the Vendor name used for the Threat Name?
The AV product name? It's quite hard because the threat name can be edited, and thus we'll loose that information

Quote
Also can someone who studies malware as a hobby can have lower subcription? $50 a month is too much for me.. When all is well and MRF, Cuckoo, and Viper working well together, I'll be deploying this to a Linux 2U rack and Subscibe if the price is reasonable for me..

A new major version will be released first quarter 2018, we planned to change that pricing a little bit to take in account such issues, thanks for asking.

Reply #5January 11, 2018, 08:31:36 PM

RazorBurn

  • Newbie

  • Offline
  • *

  • 8
  • Reputation:
    0
    • View Profile
Re: Need Advice on Cron Job
« Reply #5 on: January 11, 2018, 08:31:36 PM »
Disabling Cuckoo in config.php gives blank results..


Reply #6January 12, 2018, 02:33:25 PM

Tigzy

  • Administrator
  • Hero Member

  • Offline
  • *****

  • 954
  • Reputation:
    90
  • Personal Text
    Owner, Adlice Software
    • View Profile
    • Adlice Software
Re: Need Advice on Cron Job
« Reply #6 on: January 12, 2018, 02:33:25 PM »
Hey,
Do you have apache errors when doing so?

Reply #7January 13, 2018, 06:00:43 PM

RazorBurn

  • Newbie

  • Offline
  • *

  • 8
  • Reputation:
    0
    • View Profile
Re: Need Advice on Cron Job
« Reply #7 on: January 13, 2018, 06:00:43 PM »
from
Code: [Select]
/var/log/apache2/error.log
Cuckoo Enabled in config.php
Code: [Select]
[Sun Jan 14 00:50:41.748166 2018] [mpm_prefork:notice] [pid 67] AH00163: Apache/2.4.18 (Ubuntu) configured -- resuming normal operations
[Sun Jan 14 00:50:41.748370 2018] [core:notice] [pid 67] AH00094: Command line: '/usr/sbin/apache2'
[Sun Jan 14 00:54:10.804327 2018] [:error] [pid 72] [client 127.0.0.1:62320] PHP Notice:  Only variables should be passed by reference in /var/www/html/mrf/src/modules/cuckoo.php on line 25
[Sun Jan 14 00:54:22.994554 2018] [:error] [pid 544] [client 127.0.0.1:62330] PHP Notice:  Only variables should be passed by reference in /var/www/html/mrf/src/modules/cuckoo.php on line 25, referer: http://localhost/mrf/index.php
[Sun Jan 14 00:54:25.060991 2018] [:error] [pid 544] [client 127.0.0.1:62330] PHP Notice:  Only variables should be passed by reference in /var/www/html/mrf/src/modules/cuckoo.php on line 25, referer: http://localhost/mrf/index.php

Cuckoo disabled in config.php
Code: [Select]
[Sun Jan 14 00:58:33.989768 2018] [mpm_prefork:notice] [pid 612] AH00163: Apache/2.4.18 (Ubuntu) configured -- resuming normal operations
[Sun Jan 14 00:58:33.990017 2018] [core:notice] [pid 612] AH00094: Command line: '/usr/sbin/apache2'
[Sun Jan 14 00:58:37.802995 2018] [:error] [pid 619] [client 127.0.0.1:62375] PHP Deprecated:  Methods with the same name as their class will not be constructors in a future version of PHP; floIcon has a deprecated constructor in /var/www/html/mrf/src/lib/ico/floIcon.php on line 222
[Sun Jan 14 00:58:37.803371 2018] [:error] [pid 619] [client 127.0.0.1:62375] PHP Deprecated:  Methods with the same name as their class will not be constructors in a future version of PHP; floIconImage has a deprecated constructor in /var/www/html/mrf/src/lib/ico/floIcon.php on line 356

Thanks for looking into it.. Then again this is running on Windows Subsystem for Linux on Windows 10 build 1709 x64bit.. Can't find any fault aside from this one, even Cuckoo running on a separate system is working fine.. Disabled the Cuckoo cause I have some few problems with it connecting the VM to the Host, but thats a problem I have to resolve within Cuckoo, but Cuckoo part in MRF is working fine..
« Last Edit: January 13, 2018, 06:19:32 PM by RazorBurn »

Reply #8January 15, 2018, 03:59:28 PM

Tigzy

  • Administrator
  • Hero Member

  • Offline
  • *****

  • 954
  • Reputation:
    90
  • Personal Text
    Owner, Adlice Software
    • View Profile
    • Adlice Software
Re: Need Advice on Cron Job
« Reply #8 on: January 15, 2018, 03:59:28 PM »
Hey, are you running PHP7?
Just checking because it's not supported yet.

Reply #9January 16, 2018, 01:12:37 AM

RazorBurn

  • Newbie

  • Offline
  • *

  • 8
  • Reputation:
    0
    • View Profile
Re: Need Advice on Cron Job
« Reply #9 on: January 16, 2018, 01:12:37 AM »
 
Hey, are you running PHP7?
Just checking because it's not supported yet.

Might be that.. but php7 is the default php in WSL Ubuntu 16.04.03.. Well try to downgrade to PHP5 and test..

Reply #10January 16, 2018, 02:10:28 PM

Tigzy

  • Administrator
  • Hero Member

  • Offline
  • *****

  • 954
  • Reputation:
    90
  • Personal Text
    Owner, Adlice Software
    • View Profile
    • Adlice Software
Re: Need Advice on Cron Job
« Reply #10 on: January 16, 2018, 02:10:28 PM »
Yes but it wasn't tested yet. We know some people got error with it, so it's not surprising.
It's scheduled though, will be in next version. We'll also consider publishing a Docker image

Reply #11January 16, 2018, 06:13:26 PM

RazorBurn

  • Newbie

  • Offline
  • *

  • 8
  • Reputation:
    0
    • View Profile
Re: Need Advice on Cron Job
« Reply #11 on: January 16, 2018, 06:13:26 PM »
Yes but it wasn't tested yet. We know some people got error with it, so it's not surprising.
It's scheduled though, will be in next version. We'll also consider publishing a Docker image

Thanks, Docker Image is huge + for me. Save me time setting up MRF. Focusing instead in Cuckoo, Viper, & other tools.

Reply #12January 17, 2018, 07:15:13 AM

RazorBurn

  • Newbie

  • Offline
  • *

  • 8
  • Reputation:
    0
    • View Profile
Re: Need Advice on Cron Job
« Reply #12 on: January 17, 2018, 07:15:13 AM »
Any Idea how to use the virustotalscan API in MRF. A sample script maybe like the upload script examle  in MRF documentation.

Code: [Select]
POST
http://localhost/api.php?token=my_token&action=virustotalscan
parameters:
hash=the_md5
return data: HTTP code 200 if success

Limiting to only 4 request per minute for VT Public API..

Reply #13January 18, 2018, 09:45:02 AM

Tigzy

  • Administrator
  • Hero Member

  • Offline
  • *****

  • 954
  • Reputation:
    90
  • Personal Text
    Owner, Adlice Software
    • View Profile
    • Adlice Software
Re: Need Advice on Cron Job
« Reply #13 on: January 18, 2018, 09:45:02 AM »
Hey, I guess something like this should work (untested)

Code: [Select]
import json
import os
import requests
 
# Parameters, don't forget to modify
apikey      = "your_token"
host        = "mrf.yourserver.com"
urlserver   = "http://mrf.yourserver.com/api.php?action=virustotalscan"
md5         = "the_md5_of_the_file_already_uploaded"
 
def post_multipart(host, selector, fields):
    headers = {'user-agent': 'Dionaea honeypot'}
    r = requests.post(selector, headers=headers, data=fields)
   
def ScanFile():                 
    parameters = {"hash": md5}
    post_multipart(host, urlserver, parameters)