Have you ever wondered how different apps or websites can communicate with each other and share information? Well, that's the magic of APIs. APIs, or Application Programming Interfaces, are like a set of rules and protocols that allow different software programs to talk to each other and share data or functionality. For example, when you use a weather app on your phone, it's probably using an API to get the current weather data from a service. Or when you log into a website using your Google account, that's also using an API to access your account information from Google. APIs are everywhere around us and they make software development a whole lot easier.
In this article, you will learn about:
- how APIs work?
- why they are API important?
- what are the different kinds of APIs?
- what is an API integration?
- how as a developer you can make use of API?
Why are APIs important?
You just learned that APIs are a set of rules and protocols that help different softwares or even different pieces of software to communicate with each other. This might naturally lead you to wonder why APIs are even important or required.
As it turns out, APIs are integral in letting other developers use your applications. Let's try to understand this with an example. You might have seen different websites that implement the "Sign in with Facebook" or "Sign in with Google" option. Something similar to how ScrapingBee implements them:
These buttons make use of a protocol called OAuth which defines how websites/apps like ScrapingBee can get user data from Facebook and Google for a particular user after the user grants them access. Facebook and Google have created an API that implements the OAuth protocol. 3rd party websites like ScrapingBee do not need to know how Facebook and Google perform authentication. They just need to know how to request Facebook or Google to authenticate a user on their behalf and how Facebook and Google can inform them of a successful authentication. Without any proper API by these platforms, it would have been very difficult to implement social login functionality into applications.
There are countless similar examples. Look at your computer. You can plug whatever mouse or keyboard you want into your computer and it just works. This is possible with the power of well-defined APIs. There is an agreed-upon protocol and method of communication which all mice and keyboards have to adhere to if they want to communicate with the Operating System. Due to this "API", new vendors can create new peripherals that are compatible with almost all widely used Operating Systems out-of-the-box.
Different kinds of APIs
As you can imagine from the loose description of APIs, there are countless APIs in existence and new ones are showing up every day. Let's discuss a few important kinds of APIs and how they function.
What is a REST API?
If you are making an app that relies on data from a 3rd party online service, you must have come across REST (Representational state transfer) APIs. This is the most common way of implementing APIs on the web. A REST API is based on the HTTP protocol and uses HTTP requests to POST (create), PUT (update), GET (read), and DELETE data. These APIs are often exposed over a simple URL like https://example.com/api/users and applications can interact with this API by making HTTP requests to this URL/endpoint.
This architectural pattern for creating APIs was defined by Dr. Roy Fielding in his 2000 doctorate dissertation. He laid out 6 characteristics that an API needs to conform to be considered a REST API.
The REST APIs generally communicate with JSON data. No specification requires the request/response data to be in JSON but this became a convention as JSON can easily be parsed in almost any language.
An example of a REST API is the Free Dictionary API. You can use this API by requesting https://api.dictionaryapi.dev/api/v2/entries/en/<word> after replacing <word> with any word you want to look up in the dictionary. You will get a similar response if you try to look up "hello":
[ { "word": "hello", "phonetic": "həˈləʊ", "phonetics": [ { "text": "həˈləʊ", "audio": "//ssl.gstatic.com/dictionary/static/sounds/20200429/hello--_gb_1.mp3" }, { "text": "hɛˈləʊ" } ], "origin": "early 19th century: variant of earlier hollo ; related to holla.", "meanings": [ { "partOfSpeech": "exclamation", "definitions": [ { "definition": "used as a greeting or to begin a phone conversation.", "example": "hello there, Katie!", "synonyms": [], "antonyms": [] } ] }, { "partOfSpeech": "noun", "definitions": [ { "definition": "an utterance of ‘hello’; a greeting.", "example": "she was getting polite nods and hellos from people", "synonyms": [], "antonyms": [] } ] }, { "partOfSpeech": "verb", "definitions": [ { "definition": "say or shout ‘hello’.", "example": "I pressed the phone button and helloed", "synonyms": [], "antonyms": [] } ] } ] }]What is a SOAP API?
SOAP APIs are another type of web API. They use a different protocol called SOAP (Simple Object Access Protocol) to send and receive data. SOAP is an XML-based protocol, which means it uses a specific format for the messages that are sent back and forth between the client and the server.
SOAP APIs are typically used for more complex, enterprise-level applications, and they often require more setup and configuration than REST APIs. They can be a bit trickier to work with than REST because the messages are in XML format, which can be more difficult to parse and understand.
Here is what a sample SOAP request for a coupons API might look like:
POST /Coupons HTTP/1.0Host: www.coupons.orgContent-Type: text/xml; charset = utf-8Content-Length: xxx<?xml version = "1.0"?><SOAP-ENV:Envelope xmlns:SOAP-ENV = "http://www.w3.org/2001/12/soap-envelope" SOAP-ENV:encodingStyle = "http://www.w3.org/2001/12/soap-encoding"> <SOAP-ENV:Body xmlns:m = "http://www.xyz.org/quotations"> <m:GetCoupon> <m:CompanyName>ScrapingBee</m:CompanyName> </m:GetCoupon> </SOAP-ENV:Body></SOAP-ENV:Envelope>And here is what a corresponding response might look like:
HTTP/1.0 200 OKContent-Type: text/xml; charset = utf-8Content-Length: xxx<?xml version = "1.0"?><SOAP-ENV:Envelope xmlns:SOAP-ENV = "http://www.w3.org/2001/12/soap-envelope" SOAP-ENV:encodingStyle = "http://www.w3.org/2001/12/soap-encoding"> <SOAP-ENV:Body xmlns:m = "http://www.xyz.org/quotation"> <m:GetCouponResponse> <m:Coupon>Here is the quotation</m:Coupon> </m:GetCouponResponse> </SOAP-ENV:Body></SOAP-ENV:Envelope>What is an Operating System API?
Similar to how SOAP and REST APIs allow two applications to talk to each other, there are also operating system APIs that facilitate communication between a process and the underlying hardware.
For example, let's say you're writing a program that needs to access a file on your computer's hard drive. Instead of having to write code that directly interacts with the hard drive, you can use the operating system's API to perform file operations. This way, you don't have to worry about the details of how the hard drive works, and you can focus on the logic of your program.
These operating system APIs are generally implemented via system calls. An example might be the open() function in the standard C library. It makes use of the lower-level open system call and lets you open a file without worrying about how to seek to the specific sector of the hard drive where the file might be located.
What is a Database API?
Database APIs allow you to access a database and the information within it without knowing the specifics of how the database works. This is how you can access and manipulate the same database from different programming languages like Python and Go. The database is not re-implemented in all these different languages but rather it exposes an API with some instructions on how to communicate with it. This API is then used to create libraries and packages in different programming languages for the database.
A database API usually exposes all the operations you might be able to do natively on the database. This involves adding, deleting, updating, and reading database records. There are different types of famous database APIs, like ODBC, JDBC, ADO.NET, and others. Most of them are tailored to a specific programming language, or a specific database management system.
What is an API integration?
An API integration means connecting two or more applications via their APIs. When you see web applications advertising about their integrations with other applications/websites, they are doing so via API integrations. An example of this might be how you can receive a message in Slack whenever someone creates a new Jira issue. Jira has integrated with the Slack API and they create a new message whenever your Jira receives a new issue. Alternatively, Jira also allows you to create a new issue directly from Slack.
APIs vs website interfaces
It is possible to use the website interfaces of different applications to get access to underlying data. For example, you can make use of Google Translate's website interface to translate text between multiple languages. However, there are a few caveats when you want to automate this interaction.
Website interfaces tend to change with little to no communication. If you are integrating with an application's website interface using web scraping then your integration might break with a slight change in the website interface. Whereas APIs are generally versioned and a new API version does not get rid of the old API.
Moreover, APIs typically provide data in a format that is easier to process, and more often than not this format is JSON. This makes sure you do not have to spend extra effort in converting the data to a usable format.
Unless you have a good reason to use a website interface for integration, you should rely on official APIs as they are more robust and are there for this exact purpose.
How do APIs work?
Let's focus on REST APIs as they are the most common kind of APIs that you will probably encounter. An important thing to note is that REST is an architecture and not a protocol or a standard. There are different ways developers can implement REST APIs. The actual request-response cycle of the REST API will differ depending on which language and web framework you use to implement it. However, the general cycle will look something like this:
- Request: The client sends a request to the server using an HTTP method, such as GET, POST, PUT, DELETE, etc. The request includes information about what the client wants to do, such as retrieve data, create a new resource, or update an existing one.
- Routing: The server's REST API framework receives the request and maps it to a specific endpoint. The endpoint is a URL that corresponds to a specific resource or action on the server.
- Processing: The server performs the requested action and generates a response. The response includes information about the result of the operation, such as data, status codes, and headers.
- Response: The server sends the response back to the client. The response includes information about the result of the operation, as well as any data that the client requested.
- Processing: The client receives the response and processes it. Depending on the nature of the request and the response, the client may take additional action, such as displaying the data to the user or making another request to the server.
Wherever you read the word "resource" while reading about REST, it refers to data on the server. If we take a look at our previous dictionary example again, the dictionary contains a bunch of entries, and these entries are a resource. REST endpoints are generally resource oriented and one endpoint manipulates only one type of resource. Therefore, the URL for manipulating the "entries" resource explicitly contains the word "entries": https://api.dictionaryapi.dev/api/v2/entries/en/<word>.
There are four main types of requests in REST APIs. These types are based on the pre-existing HTTP verbs:
- GET: Used to retrieve information from the server. A GET request should not modify the state of the server and is typically used to retrieve a representation of a resource.
- POST: Used to send information to the server. A POST request is typically used to create a new resource or update an existing one.
- PUT: Used to update an existing resource. A PUT request should completely replace the current representation of a resource with the one specified in the request.
- DELETE: Used to delete a resource. A DELETE request should remove the specified resource from the server.
Public APIs generally very visibly highlight the type of request that is possible on each resource. Here is an example from a random API website:
As REST operates over HTTP, HTTP headers become very important as well. The headers usually contain important information related to the request as part of the following header entries:
- Accept: Specifies the media types that the client is willing to accept in the response. This is mostly JSON for REST requests.
- Content-Type: Specifies the media type of the request body, if there is one. This header is usually used in POST and PUT requests, where the client is sending data to the server.
- Authorization: Contains authentication information, such as a token or username and password. This header is used to identify the client and determine if it has the necessary permissions to access or modify the requested resource.
- Cache-Control: Specifies caching instructions for the response, such as whether the response can be cached, and for how long. Caching is an integral part of REST architecture.
- If-Modified-Since: Specifies a date and time, used by the server to determine if the client's cached copy of the resource is up-to-date. If the resource has not been modified since the specified date, the server can return a 304 Not Modified response instead of the full resource.
These header keys/values are not REST specific but rather REST makes use of these pre-existing HTTP headers to facilitate its architecture.
Here is what a typical POST REST request might look like:
Using APIs in the real world
Many popular APIs are used by developers every single day. A major one is the Google Maps embed API. Whenever you see Google maps embedded in a website, the developers are using this API in the background to do so.
Such public APIs are generally easy to use. The first step you have to follow is to go to the developer docs website for the relevant API. The docs usually provide ample information to get started with an API as quickly as possible. This is done to ensure developers don't have to spend a ton of time learning a new API.
Just to show you how easy it is to use the Maps Embed API, you just need to include the following IFrame in the body tag of your website:
<iframe width="600" height="450" style="border:0" loading="lazy" allowfullscreen referrerpolicy="no-referrer-when-downgrade" src="https://www.google.com/maps/embed/v1/place?key=API_KEY &q=Space+Needle,Seattle+WA"></iframe>The IFrame will start working once you replace API_KEY with a working API key from Google. You can request the API key by creating a developer account. This process is detailed on the API documentation website. This also gives a nice introduction to how some public APIs use API keys to make sure no unauthorized access is taking place.
Once the embed starts working, you will see a nice map of Seattle.
Some other popular APIs include the Recaptcha API. This is what shows up whenever you are browsing the internet and the website asks you to prove you are a human. This image might help you recall some fun memories when it was especially hard to prove you aren't a robot 🤓:
A very nice resource for getting information about public APIs is RapidAPI. Their website contains information about the various free and paid APIs. Another similar resource is this GitHub page.
Conclusion
In this article you learned about APIs; what they are, where to find them, how to use them, and how the different kinds of APIs differ from each other. It is hard to grasp this knowledge without experimenting. You need to try out different APIs, read their documentation, and figure out how they work.
There are a ton of tools out there that simply combine and build on top of various public APIs and have collectively raised billions of dollars from investors while doing so. Some prominent examples are the various chatbots and AI-based start-ups that are popping up each day and making use of OpenAI's API. Product Hunt is full of such examples. I am sharing this just to encourage you to go out there and explore what APIs are available and how you might be able to creatively make use of them.
And if you happen to be in the business of web scraping, it will be criminal not to mention our very own freemium API 😉. ScrapingBee offers a very easy-to-use API for scraping websites and making sure you do not have to endlessly solve captchas or face IP bans. Go through our documentation and try making a simple web scraper using ScrapingBee in your language of choice.
