I scrape pages for one of my projects, and the best way to get around anti-scraping is to occasionally fill out captchas’ / do some browsing by hand to get through the testing (usually works for a good while).
Anyway I ran into a heck of a rabbit hole actually trying to tell selenium to use a specific profile, but then I finally ran into a post that solved it for me.
There are two options you need to use a custom profile (or even the default profile).
–-profile-directory=
and
--user-data-dir=
The key is the user-data-dir is specifying where the profiles are stored on the disk, not a path to the profile but a path to the directory that has the profiles.
The --profile-directory settings is saying “use this profile” in that directory. So if your profile as re stored at /home/<userid>/.config/google-chrome then you would set --user-data-dir=/home/<userid>/.config/google-chrome.
If the profile you wanted to use was the main one you would set --profile-directory=Default.
You can see what your profile directory is by going to chrome:://version. There is a “Profile Path” variable, just make sure to remove the profile name from the path, and no trailing slash.
This is what my chrome options array looks like (ignore that I am using a custom class to init this as I have a bank of machines used for selenium and I do not have grid setup).
