Find all the duplicate photos on your Qnap with findimagedupes

You may have a lot of duplicate images on your hard drives. Now Exact duplicate images are easy to find and delete with fdupes, but what about images that are different size, cropped, or different just a bit visually ?
findimagedupes is a command line tool that finds visually similar images, even if they are of different resolution and size. There is a threshold setting to define how much similar two images have to be.
Install findimagedupes with  apt-get install findimagedupes
The simplest form to run is
findimagedupes -R ~/images
this finds all the duplicate images recursively in all subdirectorie of ~/images.
A more complete form which I generally use is

findimagedupes -v=fp  -R -f=fp_data  ~/images</div>

  • -v  : verbose mode. You can see fingerprint of eachfile with fp, or md5 hash with md5.  For some reason the option -v=LIST specified in the command help doesn’t work. You only can get -v=md5 or -v=fp to work.
  • -R  : recursively search all subdirectories
  • -f   :  This is a file to we are going to write to / or read from , each images fingerprint . especially useful for large directories with lots of images.
Generally I would let it run first for sometime so that I have all the files scanned and their signatures saved in fp_data.  On next run, the program reads off the fp_data file and runs a lot more faster.
To use a photo viewer like feh, use the -p switch with full path . This is what you should do if you want to see the pairs and delete one or more of them.  Feh lets you navigate in each set , delete the images and then move on to the next set.
-p doesn’t work with simple program name, even if this exist in your system path. You have to always specify the full path of the program.
findimagedupes -v=fp  -R -f=fp_data  -p= /usr/bin/feh ~/images
You can also set the threshold. To find Exactly same visual image, set the threshold to 100 for example
findimagedupes -v=fp  -R -f=fp_data  -p= /usr/bin/feh -t=100 ~/images

 

Couchpotato and TBP ” No ThePirateBay proxies left, please add one in settings, or let us know which one to add on the forum.”

Problem : Couchpotato doesn’t snatch any file from TBP.

Error message is

ERROR[se.providers.torrent.base] No ThePirateBay proxies left, please add one in settings, or let us know which one to add on the forum.

Solution : Add to proxy and change the search url

You have 2 python files for thepiratebay in Couchpotato.

/opt/CouchPotatoServer/couchpotato/core/media/_base/providers/torrent/thepiratebay.py
/opt/CouchPotatoServer/couchpotato/core/media/movie/providers/torrent/thepiratebay.py

We need to modify the first one inside  media/_base/providers/torrent/

You need to modify at 2 places

  1. Add the url to tbp to proxy_list

This is the list of all the urls that the website can be accessed from. at my location today I can access tbp through https://thepiratebay3.org, so I have to add it to the list of Proxies

proxy_list = [
'https://pirateproxy.cat',
'https://pirateproxy.wf',
'https://pirateproxy.tf',
'https://urbanproxy.eu',
'https://piratebays.co',
'https://pirateproxy.yt',
'https://thepiratebay.uk.net',
'https://thebay.tv',
'https://thepirateproxy.co',
'https://theproxypirate.pw',
'https://arrr.xyz',
'https://tpb.dashitz.com',
'https://thepiratebay3.org'
]

Make sure that you add a comma after the last entry , and use the space bar to indent it just below the last entry since python uses indented code.

2. update the Search url :

The Default search url in this file is websiteroot/search/search_term

however, when I search , I see that that the search url should be websiteroot/index.php?q=search_term.

So in the following code change the search term from 'search': '%s/search/%%s/%%s/7/%%s' , to  'search': '%s/index.php?q=%%s/%%s/7/%%s'

class Base(TorrentMagnetProvider):
urls = {
'detail': '%s/torrent/%s',
'search': '%s/index.php?q=%%s/%%s/7/%%s'
}

That’s it , restart your couchpotato and it should start working

Recover Data on Qnap with Photorec

Last week I got a qnap from a friend where multiple hard drives failed and he was not able to see the volumes with Qnap disk manager.  Having exhausted all the other options, we were willing to take up recovery.
The Tool that came to my mind first was photorec. Its open source, runs on Linux ( so I can run it directly on Qnap), and mounts your hard drive in read only mode so you are not making the situation any worse by using this.

Step 1 :  Get Putty and Photorec

You will need to get to the Qnap command prompt. Download Putty if you are on windows and connect to your Qnap using putty.
if you run uname -a on your qnap it will tell you that you are running kernel 4.2 so you can use photorec which needs kernel 2.6 or later
You will also need Photorec . Download and save it on an external hard drive/flash or on a network share that you can mount from Qnap you are recovery data on.

Step 2 : Prepare your media where the recovered files are to be saved

If you have a large External Hard drive, you can save your Photorec on this  and save the recovered files on this too.
If not, you can mount any network drive in your Qnap and save files on that. This is what I did for my friends Qnap as his HDD size was 6TB and I couldn’t find an external hard drive large enough to fit all the data.
You can mount any other qnap or windows share by first creating a directory and then mounting your share to this directory. You need ato save data on a drive other than the one you are recovering from.
mkdir /mnt/recovery
mount -t cifs -o username= <yourusername>  <ipaddress/path_to_sharedfolder>  /mnt/recovery

Step 3 : Run photorec recovery and wait for it to finish.

Insert your External Storage into the USB slot of Qnap and check if this is detected and the device name this is given.
cd /share/external
ls
here you will see some device with DEVXXXXX , this is your external drive. cd to this directory, your photorec directory and run photorec_static
If you get permission denied, you would have to change it to executable first.
chmod 777 photorec_static
run it by running ./photorec_static
Rest is all wizard based, you will have to select your hard drive , or your raid and select where to save the files.
your single hard drives will be named  sda, sdb, sdc etc
Raids will be md0,md1 etc
Thin Volumes will be cachedev1, cachedev2 etc
What is important is to select your external drive ( /share/ext/DEVXXX)  or your mounted network drive ( which we mounted to /mnt/recovery) when it asks you where you want to store the recovered files to.  DO NOT STORE THEM ON YOUR EXISTING VOLUME

Optional : Use Screen Command to Run the Wizard so you can close your putty session and let is run in the background

Screen is preinstalled on Qnap and I like to use it for running commands that take long time.
Run screen it takes you to a new terminal where you can run all your commands. Run all the commands of step 3 in this terminal.
To return to your original terminal, press Control +A  , then press D ( as in Detach). This will detetch the screen and you will be back to the main prompt
Though you deteched from that screen, you didn’t kill it, so your process is running in that screen. To go back to the screen, type Screen -r
It will either show you a list of all the screens that are available to attach to , or reattach you to the screen if there is only 1.  It showed me the following output

There are several suitable screens on:
16993.pts-2.AJQNAP      (Detached)
19381.MYTRANSCODE       (Detached)

You can now to go your screen by typing screen -r 16993 ( the number before the dot) and it takes you to your screen.

Limitations of Data Recovery this way

  1. It doesn’t restore your directory structure.
  2. It doesn’t restore your file names
  3. Some files Meta data ( exif data in pictures for exmaple) may not be recovered
  4. Your data will be spread in several directories. I had 12,000 directories called recup_dir.xxxx
  5. There will be several possible duplicates. you can use findimagedupes on linux or a tool like Visipics on windows to elimiate duplicates

Install Transmission on Qnap with Docker the Easy way

Transmission is the favorite BT client for many , but Qnap doesn’t offer it in app center any more.  However Qnap now offers you container station in which you can run LXC or Docker container.  With Docker, you can very easily install transmission and several other applications Graphically , within minutes

Step 1.  Find The docker Image

  1. Click on Container Station
  2. Click on Create
  3. Search for “Transmission”
  4. Select the image from Linux Server
  5. Click on Create
  6. Enter the Name of your Container. Auto Start is selected by Default
  7. Set the CPU and Memory to half of what you have. This container takes very little resources ( less than 1% of CPU and RAM in my case) , but you can set the upper limit to 50% .

Step 2.  Set the Network to Bridge Mode ( with DHCP or Static)

You will need an IP address to access your transmission webfront. For this set the network on Bridge mode. Set it to DHCP if you have a DHCP server,or Static if you want to have a Fixed IP.

Step 3.  Specify your Config, Download and Watch Directory

You need to mount 3 directories.

/config  holds your configuration files.
/downloads  is where your downloads are saved
/watch is where you place new torrents.

You can skip mounting the config directory and it will by default create one for you.

For /downloads you would always want to mount a directory on Qnap.  I use /Download directory on qnap for downloading so I specified this. Further I created watch and config directories under download .

Step 4. Run the container, find the ip and access transmission webfront

Run the container and you can see its IP address in the console. In case you don’t see the IP address, you can click on terminal and run ifconfig command. It will open a new window with the output and you will have the ip address.

Use that ip address to access your transmission webfront with the port 9091.  ( for example 192.168.1.198:9091 )

There is no username or password required to login. If you want further security you can edit the /config/settings.json file to specify which ip address are allowed to login from, a username and a password

That’s all there is to running transmission on your Qnap. In next tutorial I will show you how to install couchpotato using Docker container on qnap.

Running Couchpotato on Qnap with Container Station

The Easiest Way to run Couchpotato on Qnap is with Container Station   The Easiest way is to use Docker on Container station, in this tutorial we used an LXC Container with full Ubuntu installation. We can instead use lightweight Docker container.

1. Create a Ubuntu Container on your Qnap

  • Open Container station on Qnap and click on create.
  • Search for Ubuntu Image
  • Click on Install
  • Adjust your CPU and Memory Settings
  • Set Your Network on Bridge Mode ( DHCP or Static)

This is your container in which you will install Couchpotato. Its a python script to first you will need to have python on your system. It needs python 2.7 ( not python 3).

2. Install Python on your container

Install python by running  these commands inside your container.  default username and password are ubuntu:ubuntu , which you are reminded in the terminal window .

sudo apt-get update
sudo apt-get install python

check if you got the correct version by running python --version. it should report 2.7.x

3. Install git on your container and Clone Couchpotato

You will use git to get couchPotato to your container, so you will need to have git installed on your container first. do it by running

sudo apt-get install git-core

Once Git is installed you can clone couchpotato.  go to any directory you wish to save the couchpotato in ( I use /usr/local/bin) and run

sudo git clone git://github.com/CouchPotato/CouchPotatoServer.git

4. Run Couchpotato and Access its webfront 

To run Couchpotato you can run

python  /usr/local/bin/CouchPotatoServer/CouchPotato.py 

You can access the webfront in any webbrowser by visiting <containerip>:5050

In the next article I will post how to install Transmission on Qnap using Container station. Your Couchpotato can use transmission to download files.