In the evolving landscape of data integration, staying ahead means continuously enhancing the versatility and effectiveness of our tools. That’s why we’re thrilled to announce the latest addition to Conduit: the HTTP Connector. This powerful plugin not only broadens the scope of Conduit’s capabilities but also simplifies the process of pulling and pushing data over HTTP.
Why the HTTP Connector?
A generic HTTP connector allows you to connect to any HTTP-based service or API. This flexibility is essential in modern software development, where systems often need to communicate with a wide range of external services, from internal APIs to third-party platforms.
By having both the HTTP source and destination connectors, you gain the ability to effortlessly transfer data between any Conduit source connector to an HTTP endpoint, and vice versa.
Building and Testing Made Simple
Developed with ease of use in mind, building and testing the HTTP Connector is straightforward:
- Download or Build: Download the connector’s release binary file that is ready to use, or use the simple
make buildcommand that compiles the connector from source, preparing it for integration into your Conduit pipeline. - Testing: With
make test, you can run through all the unit tests to ensure the connector functions correctly.
Source Connector: How It Works
The source side of the HTTP Connector pulls data at regular intervals specified by the pollingPeriod. It’s smartly designed to enhance flexibility, allowing you to specify request methods (GET, HEAD, OPTIONS), headers, and parameters to tailor the data request to your needs. Particularly noteworthy is the use of the OPTIONS method, which appends the returned options directly to the record's metadata, enriching the data ingested into Conduit.
Configuration options:
- URL: The endpoint from which data is fetched.
- Method: Choose from GET, HEAD, or OPTIONS to match the endpoint’s requirements.
- Headers and Params: Further customize your requests.
- Polling Period: Set how frequently the connector fetches data, with a default of every 5 minutes.
Example:
Using the HTTP source connector, you can pull data from any HTTP API and push it to any Conduit destination connector, check our Conduit connectors list. Let’s take for example a pipeline that pulls orders from Shopify and pushes them into a file destination connector:
Pipeline Configuration File
Create a folder called pipelines at the same level as your Conduit binary. Inside of that folder create a yaml file and copy these configurations over.
version: 2.2
pipelines:
- id: shopify-pipeline
status: running
connectors:
- id: shopify-orders
type: source
plugin: standalone:http
settings:
url: <https://cd8206-5c.myshopify.com/admin/api/2024-04/orders.json> # your shopify API to get orders.
headers: X-Shopify-Access-Token:${SHOPIFY_ACCESS_TOKEN} # reference to an env-var that has your access token.
pollingPeriod: 30m # pull data from the URL every 30 minutes.
- id: file-dest
type: destination
plugin: builtin:file
settings:
path: orders.txt
processors:
- id: decode-response
# use a builtin processor that decodes the pulled data into JSON.
plugin: json.decode
settings:
field: .Payload.After
Now run Conduit using ./conduit, and see the magic!
This pipeline will pull the Shopify orders from the API every 30 minutes, parse the response into JSON, and then write the orders into the destination file.
Check Pipeline Configuration Files for more details around the pipeline configuration files and how to run them.
Destination Connector: Pushing Data Forward
On the flip side, the destination connector takes data processed in Conduit and pushes it to the specified HTTP endpoint. Like its source counterpart, it allows for detailed configuration, including request methods suitable for creating or modifying resources (POST, PUT, DELETE, PATCH). This opens up a myriad of possibilities for integrating with APIs across the web.
Configuration options:
- URL: The endpoint to which data is sent.
- Method: Supported methods include POST, PUT, DELETE, and PATCH, providing flexibility based on the API’s requirements.
- Body Manipulation: Through Conduit's built-in or standalone processors, customize the data format to fit the destination's needs perfectly.
Example:
Using the HTTP destination connector, you can pull data from any Conduit source connector and push it to an HTTP API, check our Conduit connectors list. Let’s take for example a pipeline that pulls orders from a Generator source connector and pushes them into the Shopify API to add products:
Pipeline Configuration File
Create a folder called pipelines at the same level as your Conduit binary. Inside of that folder create a yaml file and copy these configurations over.
version: 2.2
pipelines:
- id: shopify-pipeline
status: running
connectors:
- id: generator-src
type: source
plugin: builtin:generator
settings:
format.type: structured
format.options: "title:string,body_html:string,vendor:string,product_type:string,status:string"
readTime: 1m # generate a new product every minute.
- id: shopify-products
type: destination
plugin: standalone:http
settings:
url: <https://cd8206-5c.myshopify.com/admin/api/2024-04/products.json> # your shopify API to add products.
headers: X-Shopify-Access-Token:${SHOPIFY_ACCESS_TOKEN} # reference to an env-var that has your access token.
Run Conduit using ./conduit as we did in the last example, and notice the new products generated by the source and pushed to Shopify. Check Pipeline Configuration Files for more details.
Seamless Integration with Your Data Ecosystem
The HTTP Connector is more than just a plugin; it’s a gateway to integrating a vast array of web services and APIs directly into your data pipelines. Whether you’re aggregating data from multiple sources for analysis or updating external systems with processed data, the HTTP Connector streamlines these interactions, making your data workflows more efficient and effective.
We invite you to explore the HTTP Connector and see firsthand how it can transform your data integration strategies. For more details about Conduit, visit our Conduit’s GitHub page. To get in touch join our discord channel and let us know if you have any questions.
Stay tuned for more updates as we continue to enhance Conduit, making it the most versatile and user-friendly data integration platform available. Sign up for a demo.
