To handle these situations, we're going to use Selenium. (Which I believe can be used for any programming language, but we're going to use the python version.) Selenium will help us by simulating web browser actions, such as clicking certain HTML elements. We'll also need some help from a web browser, so we're going to use Chromedriver.
Chromedriver (and Chrome) Installation
We need to install chromedriver and put it in the proper location, so that any time we call it from our python code, python knows where to find it. Thanks to this article, we can install chromedriver on Ubuntu using a few easy commands in your terminal.
This should install chrome driver, and place it in the proper folder (as well as create some symbolic links.) Again, full credits to christopher.su for the commands.
P.S. I'm pretty sure you need chrome to do this. If you don't already have chrome on your Ubuntu, then use the following commands (which I found here):
The above commands should install Chrome. (If you don't already have it.)
Our web browser is set up, now we're going to actually create a project. I find that it's best practice to use a python virtual environment to handle our libraries, and so that's what we're going to use. If you don't know how to use a python virtual environment (or if you don't have Python installed yet), then see this article quickly and get it all installed.
In your project folder, run the following commands to create our main.py file (where we'll write our python code), and to install selenium in the project folder using PVE.
Next, in our main.py while, we're going to write some test code. (Thanks to eyalfrank.com for providing this example code.)
This code will simply start the chrome webdriver and navigate to srcmake.com.
Run the python code with the following command.
You should see a popup of chrome that displays the webpage.
In this tutorial, we set up our environment for being able to use Chrome's webdriver, and create a python project that installs selenium, with some simple code to get selenium and chromedriver working.
However, there's much more that we can do with selenium to interact with webpages, from clicking buttons, parsing HTML code, and scraping data. In future tutorials we'll dive deeper into selenium's tools to build webscrapers and automatically do things from our browser.
Like this content and want more? Feel free to look around and find another blog post that interests you. You can also contact me through one of the various social media channels.