Introduction - Debugging?
Debugging is necessary for any program of any size. It's inevitable that our code will return incorrect results, and we'll want to know what values are along the way to see where things are going wrong. For Linux, "gdb" (GNU Debugger) is one tool used for debugging C and C++ code. It can be used for larger programs, but we're going to keep things simple in this tutorial by showing how to use gdp with basic commands, for a simple program.
Installation
We'll assume that you're running Ubuntu 16.04 for this gdb installation, but the requirements are very basic. (We'll also assume that you have GCC installed to compile our C++ programs.
Run the following command in your terminal to install gdb and necessary dependencies:
sudo apt-get install libc6-dbg gdb valgrind
And that's it.
A Basic C++ Program (With Some Bugs)
We're going to create a basic C++ program, and it's going to have a few bugs. We'll need to go through the code (with a debugger) to see why we aren't getting the expected result.
First of all, in your project directory, use the the following commands to create a makefile and our cpp file.
touch main.cpp touch makefile
Open "makefile" and add the following code to it:
(If you don't know about makefiles, look at this quick tutorial.)
Notice the extra -g flag for our compile command. This will tell the compiler to leave our code intact in the executable for debugging.
Our C++ program will ask the user for a number, and will calculate the factorial of that number.
Open "main.cpp" and add the following code to it.
Our code is complete.
In your terminal, run "make" to test the code out.
make
Obviously, the answer our program outputs is wrong. Why is it wrong? Well, we could inspect the code to find out, but we could also use gdb to see what's happening in our code.
Using gdb to Debug The Program
We're going to use gdb to debug our program. Let's go through the commands that we need.
First of all, start gdb. (This still start a gdb shell session.)
gdb
Next, we're going to tell gdb that our executable file (named "run") is the one we want to debug.
file run
Now we're going to set a breakpoint at main. (A breakpoint is basically where we tell our code "stop here so we can look at it".)
break main
Next, we start the program (executable) to see what's going on in the code.
run
We're going to say "next" (or "n") to go through the code line by line (as it's executed, not necessarily as it's written. For example, for loops will be called a lot).
next
At any time, to see the value of a variable, you can use "p variablename".
p factorial
This whole process will look something like this.
To quit out of the gdb shell sessions.
quit
And that's it. Use gdb to find out what's going on in your code. Conclusion
In this tutorial, we had a simple C++ program, with a few bugs. To find out what was going on in our code, we used gdb to go through the code line by line to see what the values of variables were and how the code was executing.
We did a basic tutorial, but gdb can be used for much larger examples. To proceed further, familiarize yourself with all of the gdb commands, and use them as you need to for your own program.
Like this content and want more? Feel free to look around and find another blog post that interests you. You can also contact me through one of the various social media channels.
Twitter: @srcmake Discord: srcmake#3644 Youtube: srcmake Twitch: www.twitch.tv/srcmake Github: srcmake
To watch the youtube video where I explain the process and we walk through it together, please click here.
Introduction
In the previous blog post in this series, we installed Actian Vector locally on our Ubuntu 16 virtual machine, we loaded a dataset into a Vector database, and we queried the database using basic SQL commands. Make sure to check that blog post out before doing this one.
In this blog post, we'll look at visualizing some key insights from the dataset using the fundamentals of creating a visualization dashboard. What Is a Data Visualization Dashboard?
Basically, we have some data, and we want to look at important parts of the data in some pretty picture format. A dashboard is typically an HTML page with multiple chart types, which is possibly interactive and allows the viewer to change views/data shown in the graphs.
Goals
There are several tools to assist with creating a dashboard. However, in this tutorial, we're going to focus on the actual mechanics, concepts, and programming behind making a dashboard, rather than using a tool.
Remember: we have an Actian Vector database installed locally on our computer that we're able to query. We want to query this data and present it in a nice format. So what do we have to do? Here's our task list: 1. We need to choose what data we want to visualize. 2. We need to extract this data from the database with some SQL queries. 3. We need to write a program that puts the extracted data into chart formatting. 4. We want to put the charts into a simple HTML page. What Part of the Dataset Should We Visualize?
Before going into the code and querying, let's think about this particular dataset and try to come up with some interesting insights that would be good to visualize.
The dataset is of airplane data from 1987 to 2017, and contains records from each month, for each flight. Let's try to think of at least six interesting facts that we'd want to learn from the data, as well as the chart type that would best suit that data.
1. The number of flights that were cancelled (compared to the number of successful flights). - Pie Chart
2. The season with the most flights. - Pie Chart 3. The number of flights by year. - Scatter Plot 4. The seven most popular destination states. - Bar Graph 5. The average flight time. - Just the number itself is enough. 6. The delay type that causes the most problems. - Bar Graph
Now that we know the data we want, we have to design some SQL queries to get this data.
The SQL Queries
Before doing anything., open a new terminal, and make actian a sudo user. Then log in as Actian, and start ingres.
sudo usermod -aG sudo actian su - actian source ~/.ingVWsh Let's go through each query that we need. (Note: There are cleaner SQL queries we could use, but that would make the programming a bit trickier. Let's keep it simple for this tutorial.) For each query, we'll create a .sql file that stores the SQL queries. (Black background = terminal commands to create and open the command.sql files, and white background = the SQL queries that go inside the files.)
1. We need to know the number of cancelled flights, and the number of non-cancelled flights.
touch /home/actian/command1.sql sudo gedit /home/actian/command1.sql SELECT COUNT(*) FROM ontime WHERE (Cancelled!=0);\g SELECT COUNT(*) FROM ontime WHERE (Cancelled=0);\g 2. We need to count how many flights there were in January, February, ..., December. touch /home/actian/command2.sql sudo gedit /home/actian/command2.sql SELECT COUNT(*) FROM ontime WHERE (Quarter=1);\g SELECT COUNT(*) FROM ontime WHERE (Quarter=2);\g SELECT COUNT(*) FROM ontime WHERE (Quarter=3);\g SELECT COUNT(*) FROM ontime WHERE (Quarter=4);\g 3. We need to count the number of flights for each year. touch /home/actian/command3.sql sudo gedit /home/actian/command3.sql SELECT COUNT(*) FROM ontime WHERE (Year=1987);\g SELECT COUNT(*) FROM ontime WHERE (Year=1988);\g SELECT COUNT(*) FROM ontime WHERE (Year=1989);\g SELECT COUNT(*) FROM ontime WHERE (Year=1990);\g SELECT COUNT(*) FROM ontime WHERE (Year=1991);\g SELECT COUNT(*) FROM ontime WHERE (Year=1992);\g SELECT COUNT(*) FROM ontime WHERE (Year=1993);\g SELECT COUNT(*) FROM ontime WHERE (Year=1994);\g SELECT COUNT(*) FROM ontime WHERE (Year=1995);\g SELECT COUNT(*) FROM ontime WHERE (Year=1996);\g SELECT COUNT(*) FROM ontime WHERE (Year=1997);\g SELECT COUNT(*) FROM ontime WHERE (Year=1998);\g SELECT COUNT(*) FROM ontime WHERE (Year=1999);\g SELECT COUNT(*) FROM ontime WHERE (Year=2000);\g SELECT COUNT(*) FROM ontime WHERE (Year=2001);\g SELECT COUNT(*) FROM ontime WHERE (Year=2002);\g SELECT COUNT(*) FROM ontime WHERE (Year=2003);\g SELECT COUNT(*) FROM ontime WHERE (Year=2004);\g SELECT COUNT(*) FROM ontime WHERE (Year=2005);\g SELECT COUNT(*) FROM ontime WHERE (Year=2006);\g SELECT COUNT(*) FROM ontime WHERE (Year=2007);\g SELECT COUNT(*) FROM ontime WHERE (Year=2008);\g SELECT COUNT(*) FROM ontime WHERE (Year=2009);\g SELECT COUNT(*) FROM ontime WHERE (Year=2010);\g SELECT COUNT(*) FROM ontime WHERE (Year=2011);\g SELECT COUNT(*) FROM ontime WHERE (Year=2012);\g SELECT COUNT(*) FROM ontime WHERE (Year=2013);\g SELECT COUNT(*) FROM ontime WHERE (Year=2014);\g SELECT COUNT(*) FROM ontime WHERE (Year=2015);\g SELECT COUNT(*) FROM ontime WHERE (Year=2016);\g SELECT COUNT(*) FROM ontime WHERE (Year=2017);\g 4. We need to count each destination state, order them by the ones that show up most, and then take the top seven. touch /home/actian/command4.sql sudo gedit /home/actian/command4.sql SELECT TOP 7 DISTINCT DestState AS State, Count(DestState) AS FlightsToState FROM ontime GROUP BY DestState ORDER BY FlightsToState DESC;\g 5. We need to average the value in the (arrival time - departure time) for all flights that were not cancelled. touch /home/actian/command5.sql sudo gedit /home/actian/command5.sql SELECT AVG(ArrTime-DepTime) FROM ontime;\g 6. For each delay type, we'll check if the value is non-zero or empty (meaning the delay type caused an issue). (Note: We declared this value as a VARCHAR, not INTEGER.) There are CarrierDelays, WeatherDelays, NASDelays, SecurityDelays, LateAircraftDelays. touch /home/actian/command6.sql sudo gedit /home/actian/command6.sql SELECT COUNT(*) FROM ontime WHERE (CarrierDelay != 0 AND CarrierDelay IS NOT NULL);\g SELECT COUNT(*) FROM ontime WHERE (WeatherDelay != 0 AND WeatherDelay IS NOT NULL);\g SELECT COUNT(*) FROM ontime WHERE (NASDelay != 0 AND NASDelay IS NOT NULL);\g SELECT COUNT(*) FROM ontime WHERE (SecurityDelay != 0 AND SecurityDelay IS NOT NULL);\g SELECT COUNT(*) FROM ontime WHERE (LateAircraftDelay != 0 AND LateAircraftDelay IS NOT NULL);\g Our SQL queries are all chosen, and inside of the files that we'll use to invoke them. DON'T do this yet, but the following commands are what we're going to be using to invoke our SQL commands. For each command.sql file (which holds data that we're trying to visualize with a chart), we're going to run the SQL commands inside, and output the results to a text file. sql ontimedb -S < command1.sql >/home/actian/1.txt sql ontimedb -S < command2.sql >/home/actian/2.txt sql ontimedb -S < command3.sql >/home/actian/3.txt sql ontimedb -S < command4.sql >/home/actian/4.txt sql ontimedb -S < command5.sql >/home/actian/5.txt sql ontimedb -S < command6.sql >/home/actian/6.txt
And with that, our SQL setup is done.
A Program To Format The Data Into Pictures
So we have all of the data that we're going to need, but how do we visualize it? Basically, we want to make charts/diagrams for each of our facts right now. To say it more concretely, we'll have the results of our queries in six different files, and we want to create pictures (.png files) for each. How can we do this? Any programming language will work, but let's try to keep it simple by using Python.
Create a file named "makepics.py" in your actian user's directory with the following command. The subsequent commands will install the python libraries that we need. (Make sure your terminal is still logged in as actian.)
touch /home/actian/makepics.py sudo apt-get install python-pip sudo pip install matplotlib sudo gedit /home/actian/makepics.py
Open "makepics.py" and add the following code, which will create our pictures for us. (Read through the commands to see what the code does.)
The code will read the contents of each data file, that was made with our SQL queries, and create the appropriate chart for that data. These charts will be saved as PNG images. So we have the images, but to be a "dashboard" we should format the images nicely.
Putting The Pictures In HTML Format
Okay, so our data was queried, turned into charts, and saved as PNG files. We just need to present these images. We could do this a lot of ways, but let's put them on an HTML page for now.
Since our image files are going to be the same name no matter what, we can actually write a static HTML page that displays each image! The following commands will create a file named "dashboard.html" and will open it for editing.
touch /home/actian/dashboard.html sudo gedit /home/actian/dashboard.html
Add the following HTML code to dashboard.html.
(Obviously, we could make the HTML a lot prettier and responsive, but I don't want to make this tutorial TOO detailed. Study responsive web design to make it even better. You can also add some basic CSS to make it look nicer.)
(Notice: We use some javascript to display the average time, since it isn't a picture.) So our dashboard is set up. Now how do we put everything together? Automating The Build Process
We want to run our SQL queries, then we want to run the script to make our images. But we want to automate that process so it's done automatically, so that any time we need to make a dashboard (such as when our database has updated data), that this automatically happens.
The best way to handle this is with a makefile. (We could create a bash script, but since all we need to do is run terminal commands, a makefile is easier.) Run the following command in your terminal to create a file named "makefile" and open it for editing. touch /home/actian/makefile sudo gedit /home/actian/makefile
Add the following code to the makefile:
Now cd into actian's home directory, and actually run the makefile.
cd /home/actian make
Open "dashboard.html" to see the results! (Drag the file into your web browser, or right click-> open it using your web browser.)
Conclusion
We created a data visualization dashboard, with Actian Vector as our database. Since we were working on a local machine, it was easiest to make SQL queries directly from the command line, and store those results in files. We then wrote a python script to make the resulting data pretty, in picture format, which we then visualized in an HTML page.
In further tutorials, we'll look at Actian Vector using AWS and Azure, where we can use this same visualization method. The only difference will be how we query the database.
The video where I walk through this tutorial with you is below.
Like this content and want more? Feel free to look around and find another blog post that interests you. You can also contact me through one of the various social media channels.
Twitter: @srcmake Discord: srcmake#3644 Youtube: srcmake Twitch: www.twitch.tv/srcmake Github: srcmake
References
To watch the youtube video corresponding to this article, please click here.
Introduction
Multiplayer games often have to store tons of information about the game and their players, such as details about characters and weapon stats in the game, as well as the levels, experience, and money for each player, among many other things. Some of the companies that own these games are nice enough to provide access to their databases, so that fan websites and tools can be created by independent developers and players to further enhance the game and create useful resources.
Hi-Rez Studios, the company that make the games Smite and Paladins, offers such a service: The Smite/Paladins API. Any developer that requests credentials and gets approved gets access to some documentation about the API, a valid API key, and an invite to a secret chat group with other developers.
In this blog post, we'll look at what the API offers, we'll see how to use the API, and we'll take a look at some examples websites that are made using the API. What Does The API Do?
Basically, Hi-Rez has a database with a bunch of information for their in-game numbers, as well as player information. The API lets us access that information via certain endpoints.
For example, one endpoint that we're given access to is the following: /getmatchhistory[ResponseFormat]/{developerId}/{signature}/{session}/{timestamp}/{player}
As the description states, if we supply a player's name to the endpoint, then Hi-Rez's servers will search their database for the match history of that player, format it, and send it back to us.
And the API has endpoints for a bunch of other stuff, not just match history. Of course, it's up to us, as the developers to make the best use of the API and be as creative as possible.
But what about all that other stuff? Response format? Signature? How do we actually use this "endpoint" to get a player's match history? Well...you have to write some code. Using The API (Low-Level Details)
The Paladins/Smite API works exactly as any other API does: it's a simple GET request to a special URL, that would even work in your browser if you entered it there. For a PC Paladins player, the URL for /getmatchhistory would look something like this:
http://api.paladins.com/paladinsapi.sv/getmatchhistory[ResponseFormat]/{developerId}/{signature}/{session}/{timestamp}/{player}
It's the same as before, but we added a base url in front the endpoint. So what's the deal with the other stuff?
ResponseFormat - JSON or XML. developerId - It was given to us when we got accepted. Every developer has one. Timestamp - The time the request was made. Player - The name of the player that we're stalking.
"Signature" and "session" are a bit tricky...Hi-Rez certainly didn't make this noob friendly.
Signatures
A signature is when you add the following together: developerId + method name ("getmatchhistory" in this case) + authenticationKey (given to us when we were accepted) + timestamp
And then you hash that with an MD5 algorithm. Sounds hard? Lol, yeah it is.
Sessions
Sessions are created with a special endpoint called /createsession. Basically, you open a connection with Hi-Rez's servers for 15 minutes by getting a special authentication token to be able to use other endpoints. /createsession[ResponseFormat]/{developerId}/{signature}/{timestamp}
So at least once every 15 minutes, you have to call this endpoint to get a session id variable, which you then use to be able to access the other endpoints.
Using The API (High-Level Details) + Example Code
"Okay, so we know how to use the API, but how to we actually use it??? Give an example!"
Using the API requires a programming language. You can use whatever language you want, but to be honest, NodeJS is the easiest. (WAYYYY easier than C# and Java.) Conveniently, one of the other developers wrote a NodeJS Wrapper for using some of the endpoints, which you can see here. We're going to be using it. (I'm going to assume you have NodeJS installed. If not, check out the beginning of this blog post.)
On your computer in a folder you want to work in, initiate a node project, and create a file named "config.js" and a file named "index.js" with the following commands in your terminal:
npm init touch config.js touch index.js
Next, we'll install the Paladins wrapper library to our project.
npm install paladins-api --save
Open "config.js" and add the following code:
Of course, enter your own credentials with the devId and authKey that you were given.
Next, open "index.js" and add the following code:
To be honest, this code is sort of bad and I wouldn't actually write my code like this, but it's just a simple example...
The code uses the wrapper to create a session, and then calls /GetMatchHistory on player "z1unknown". The results are saved to the file "out.txt". To run the code:
node index.js
You can view the results in out.txt. Of course, the results are ugly and you should use a tool to format the JSON to look nicer.
Of course, this was just a demonstration of one endpoint that we didn't actually do anything with. It takes creativity to parse this information and make it useful. Let's look at a few examples of people who actually use the Paladins/Smite API to do nice things.
Websites That Use the API
A few examples of websites that make use of the Paladins API are as follows:
paladins.guru - Shows player match history and generates an ELO for them.
The Better Meta - Analyzes champion/player/win performance for each patch. hirez-gql-api.now.sh/graphql - Allows for simplified API calls right in the browser. Here's an example, getting some details for a certain Paladins match. kusqt.com - Shows Paladin player loadouts. Conclusion
In this post, we saw how to use the Paladins/Smite API, how it works, and we even demonstrated using it in our own code. If you want to use the API, then feel free to request access, and try to program something cool.
Shout out to the active developers who use the API, and HirezAaron, who is the lead developer for the API that everyone pokes when something goes wrong or when we need something.
It's possible that I'll make a blog post for using the API to make a website, which would include the best practices and some architecture drawings, so keep an eye out.
Here's the youtube video for this article, if you want to see a walkthrough/talkthrough of the coding/concepts.
Like this content and want more? Feel free to look around and find another blog post that interests you. You can also contact me through one of the various social media channels.
Twitter: @srcmake Discord: srcmake#3644 Youtube: srcmake Twitch: www.twitch.tv/srcmake Github: srcmake |
AuthorHi, I'm srcmake. I play video games and develop software. Pro-tip: Click the "DIRECTORY" button in the menu to find a list of blog posts.
License: All code and instructions are provided under the MIT License.
|