Questions tagged [curl]
curl is a data transfer utility used for a multitude of network=related applications (such as scraping data).
715
questions
2425
votes
7
answers
3.1m
views
How do I make a POST request using curl?
How do I make a POST request using cURL's command-line tool?
Laurie Young
1453
votes
19
answers
2.4m
views
Getting curl to output HTTP status code?
I'm using curl at the command line on Linux to issue HTTP requests. The response bodies are printed to standard out, which is fine, but I can't see from the man page how to get curl to print the ...
- 15.3k
573
votes
12
answers
337k
views
Can I make cURL fail with an exitCode different than 0 if the HTTP status code is not 200?
I was always assuming that when curl got an HTTP 500 response it was returning an exit code that meant failure (!= 0), but that seems to be not the case.
Is there a way I can I make cURL fail with an ...
- 6,031
480
votes
21
answers
1.8m
views
How to download files from command line in Windows like wget or curl
How can I download something from the web directly without Internet Explorer or Firefox opening Acrobat Reader/Quicktime/MS Word/whatever?
I'm using Windows, so a Windows version of Wget would ...
220
votes
9
answers
638k
views
PowerShell equivalent of curl
Is there an equivalent of curl in PowerShell? Does it have some similar built-in capability or is there a 3rd party cmdlet?
- 14.5k
198
votes
5
answers
259k
views
How do you display POST data with cURL?
As an example, POSTing to a web server with the -v argument:
curl -v http://testserver.com/post -d "firstname=john&lastname=doe"
And the output
> POST /post HTTP/1.1
> User-Agent: curl/7....
- 8,797
154
votes
4
answers
135k
views
How to wget a file with correct name when redirected?
So after some time of searching on Google and Super User (and scanning man pages) I was unable to find an answer to something that (I think) should be simple:
If you go here:
http://www.vim.org/...
- 1,763
124
votes
2
answers
127k
views
Can cURL send requests to sockets?
I have a HTTP server running at /var/run/my-server.sock, and I want to test it by sending a simple request using cURL. Can this be done using cURL? Can it be done at all, or must there be a reverse ...
- 5,826
93
votes
9
answers
341k
views
Run cURL commands from Windows console
Is there a way to install cURL in windows in order to run cURL commands from the command prompt?
- 1,233
90
votes
8
answers
118k
views
How to suppress cUrl's progress meter when redirecting the output?
I'm trying to print just the verbose sections of a cURL request (which are sent to stderr) from the bash shell.
But when I redirect stdout like this:
curl -v http://somehost/somepage > /dev/null
...
- 5,756
85
votes
4
answers
251k
views
Specifying minor TLS version when using curl
Is there a way to specify curl to use a specific TLS version? Like 1.1 or 1.2? I can see only sslv3 and tlsv1 options in command help. I took latest src and compiled it with openssl 1.0.1e. Still dont ...
- 2,778
84
votes
9
answers
268k
views
Wget/cURL alternative native to Windows?
Is there a Wget or cURL type command line utility native to Windows Vista? How does it work?
- 841
78
votes
8
answers
351k
views
CURL to download a directory
I am trying to download a full website directory using CURL. The following command does not work:
curl -LO http://example.com/
It returns an error: curl: Remote file name has no length!.
But when ...
- 781
49
votes
2
answers
96k
views
How can I use curl with ::1 for ipv6 based loopback?
I tried a few ways but they seem to be confusing curl.
root@testt:~# curl localhost:8080
<h1>Hello world!</h1>
<p>Boom Bam Splat</p>
root@testt:~# curl ::1:8080
curl: (3) ...
- 722
47
votes
3
answers
202k
views
How to POST file contents using cURL?
I want to do an HTTP POST of the contents (as a string) of a local file located at path/to/my-file.txt to a URL endpoint at http://example.com/.
For example, I might want to do the following:
...
- 2,339
47
votes
3
answers
221k
views
How to send POST with body, headers, and HTTP params using cURL?
I found a lot of examples on how to use simple POST commands in cURL, but I didn't find examples on how to send full HTTP POST commands, which contain:
Headers (Basic Authentication)
HTTP Params (s=1&...
- 607
43
votes
4
answers
158k
views
how to use curl to verify if a site's certificate has been revoked?
To check if the certificate for google.com has been revoked, I tried the following command:
curl https://www.google.com --cacert GeoTrust_Global_CA.pem --crlfile gtglobal.pem -v
, but I got the ...
- 635
34
votes
6
answers
129k
views
Downloading file from FTP using cURL
I'm trying to use a cURL command to download a file from an FTP server to a local drive on my computer. I've tried
curl "ftp://myftpsite" --user name:password -Q "CWD /users/myfolder/" -O "myfile....
- 443
33
votes
3
answers
91k
views
Why can't cURL properly verify a certificate on Windows?
When I try to use Curl on windows, to retrieve an https url, I get the dreaded "connection error (60)."
The exact error message is:
curl: (60) SSL certificate problem, verify that the CA cert is ...
- 2,149
30
votes
6
answers
47k
views
Is "curl -u username:password http://example.com" secure?
Is curl -u username:password http://example.com secure?
If not, can you give a brief explanation of how someone could obtain your password?
- 561
30
votes
3
answers
27k
views
Curl local host names on Mac OS X Yosemite
I just upgraded from Mavericks to Yosemite, and now curl can't see loopback host names.
Set up a simple http server to test:
$ python -m SimpleHTTPServer
Serving HTTP on 0.0.0.0 port 8000 ...
Now I ...
- 2,016
29
votes
3
answers
42k
views
How can I pass my Windows credentials in cURL?
I'm writing a script that uses cURL to pull data from an internal web server. I can explicitly use --user combined with --ntlm and everything works well, but I can't get it to recognize whomever's ...
- 457
28
votes
1
answer
39k
views
'http" not supported or disabled in libcurl
I am trying to run a curl command and getting this error:
'http" not supported or disabled in libcurl
I looked where I have curl.exe, found it in C:\Program Files\GDAL and deleted it.
The error ...
- 12.8k
27
votes
2
answers
49k
views
Where's the ca cert bundle on OSX?
I'm debugging some curl ssl configuration stuff and was wondering where the ca cert bundle is installed on OSX.
- 373
26
votes
1
answer
20k
views
What units does curl use for bandwidth?
curl on the command-line displays progress like this:
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent ...
- 10.8k
24
votes
2
answers
44k
views
Windows 10 WSL cURL can't access localhost with error (Failed to connect to localhost port 80: Connection refused)
The issue with my WSL is the following my WSL can't make a cURL request anywhere under localhost. For this example, I have created a page that returns hello world.
My cURL looks like this (note that ...
- 341
23
votes
1
answer
16k
views
Telling cUrl not to normalize a url
When using curl to call some url, it simplifies the path in the url (for example from /a/../b to /b). How can I tell cUrl not to do so?
$ curl -V
curl 7.35.0 (x86_64-pc-linux-gnu) libcurl/7.35.0 ...
- 848
19
votes
2
answers
20k
views
Persistent retrying resuming downloads with curl
I'm on a mac and have a list of files I would like to download from an ftp server. The connection is a bit buggy so I want it to retry and resume if connection is dropped.
I know I can do this with ...
- 39.9k
19
votes
5
answers
150k
views
Get response body and show HTTP code by curl
I have endpoint which returns JSON (response body). I need get by curl the response body and process it (for example using jq). It works:
response=$(curl -s https://swapi.dev/api/people/1?format=json)
...
- 635
17
votes
4
answers
51k
views
curl http_code of 000
I have a shell script that I use to monitor loading times and response codes on my live server cluster. It runs a total of 250 iterations every 5 minutes, distributed across 10 servers and 6 sites. It ...
- 855
16
votes
2
answers
101k
views
How to get and use certificates with curl
Entering the following command
curl -k [email protected]:password https://www.example.com
Returns the following error
<HTML><HEAD><meta http-equiv="content-type" content="text/html;...
- 574
15
votes
5
answers
139k
views
How to make a Curl POST call in Windows?
I am trying to issue the following command under Windows 10:
D:\>curl -i -X POST -H "Content-Type:application/json" -d '{ "firstName" : "Frodo", "lastName" : "Baggins" }' http://localhost:8080/...
- 12.8k
15
votes
2
answers
24k
views
POSTing a file's contents with CURL
(The following is backstory, you can jump down to "Question:" if you'd like)
There's a webpage for one of my courses that accepts input text (expected to be Python code) and then gives some tokenized ...
- 327
15
votes
1
answer
14k
views
How do I remotely fetch files from redirected URLs from a terminal?
I want to fetch a tarball of this python library from terminal.
https://github.com/simplegeo/python-oauth2/downloads
However, I cannot simply call
wget https://github.com/simplegeo/python-oauth2/...
- 4,809
13
votes
2
answers
22k
views
How to convert HTML to text?
How it is possible to convert HTML to text file in Linux? For example I want to curl a query to Google, then convert the output html to text and read converted text on my terminal. I am using RHEL6.
- 271
13
votes
3
answers
30k
views
Site displays with Curl but not within the browser
I've set up a dev-enviroment on Debian 7 and installed the following services:
Apache
Bind9
I have one site that is reachable through "asix.debian.local". I've set up the zone and added NS and A ...
- 233
12
votes
4
answers
7k
views
How "legal" is site-scraping using cURL? [closed]
Recently I was experimenting with the cURL, and I found lot is possible with it. I built a small script that crawls a musical site, which plays online songs. On the way of my experiment, I found that ...
- 487
12
votes
1
answer
5k
views
How to download parts of the same file from different sources with curl/wget?
I have a quite large file hosted on five different servers.
I would like to be able to download different parts of the file from each server and subsequently concatenate the parts, in order to ...
- 308
12
votes
1
answer
12k
views
Exit bash script when curl gets a non 200 HTTP status
I have a bash script setup to perform a few curl requests
for currency in EUR INR JPY
do
curl -i --data '{"currency": "'$currency'"}' -H "Accept: application/json" -H "Content-Type: application/...
- 374
11
votes
2
answers
7k
views
bash: source from URL
In addition to my own computer, I sometimes use an Ubuntu cluster at my school. Rather than manually keep my .bashrc's in sync, I would like to make the school cluster's .bashrc source my personal ....
- 407
11
votes
1
answer
24k
views
cURL Upload file and send POST data
I have a web server running some PHP that checks for an image (curl -F 'imageName=@myimage') and it also checks the POST data for username=&password=.
When the PHP checks _REQUEST I cannot just ...
- 495
11
votes
2
answers
26k
views
How to download a private GitHub repo with no 'git' on my local machine?
What I want to do is to download private repository archive from GitHub, extract it, remove archive file and copy some directories that are inside downloaded project.
I tried to use wget but I cannot ...
- 223
10
votes
2
answers
33k
views
How do I send a digest auth request using curl?
While searching for a guide I found this example on Wikipedia
GET /dir/index.html HTTP/1.0
Host: localhost
Authorization: Digest username="Mufasa", realm="[email protected]", ...
- 542
10
votes
2
answers
20k
views
Why does sudo not work with curl?
I ran a command to download something with curl –
sudo curl roots.cx/get | sh
– and it threw a permission denied error. So I did sudo curl and I got the same error … why is that?
I was able to ...
- 4,027
10
votes
2
answers
2k
views
How can I use a command line tool like Wget to log into an OpenID site?
Specifically, I would like to be able to download certain pages from my user profile on the various Stack Exchange sites. I would, however, like to do this automatically (using a cron job), from ...
- 53.6k
9
votes
2
answers
15k
views
Downloading and extracting a .tar to a specific directory
I want to download a tar of a git repo and extract its contents to another folder.
The following line (broken up for clarity) works:
curl -L -0 nt-dotfiles.tar.gz https://api.github.com/repos/...
- 1,097
9
votes
3
answers
29k
views
SSL cert does not work with IP address for SAN
I am trying to create an SSL certificate for a server which will work internally without giving warnings in Chrome. I've created it using SAN with multiple Subject Alt Names of localhost and the IP ...
- 1,235
9
votes
2
answers
20k
views
cURL: multiple POST requests while reusing the TCP connection
If multiple URLs are passed to cURL, it reuses the TCP connection when possible.
For example:
curl -o 1.jpg http://example.com/1.jpg -o 2.jpg http://example.com/2.jpg
I need to do the same, but ...
- 1,803
9
votes
1
answer
12k
views
How to use gzip or gunzip in a pipeline with curl (For binary gz files)
I am using curl to download a file using the following command.
curl -O https://s3.amazonaws.com/dl4j-distribution/GoogleNews-vectors-negative300.bin.gz
I want to unpack it in a pipeline in the ...
- 93
9
votes
2
answers
11k
views
How can I instruct curl to reuse credentials after it followed a redirect?
I know that curl can follow redirects (the -L argument), and I know that I can provide credentials to a request (-u username:password), but I want to use both all the way down the redirect chain.
How ...
- 827