How does cURL handle URL encoding and special characters?

Handling URL encoding and special characters in cURL is a common requirement, especially when sending data containing spaces, Chinese characters, or special symbols. Proper encoding ensures requests are parsed correctly.

URL Encoding Basics

URL encoding (Percent Encoding) converts special characters to %XX format, where XX is the hexadecimal ASCII code of the character.

bash
# Common characters that need encoding
Space    -> %20 or +
&        -> %26
=        -> %3D
?        -> %3F
#        -> %23
%        -> %25
Chinese  -> %E4%B8%AD%E6%96%87

cURL Encoding Methods

1. --data-urlencode Automatic Encoding

bash
# Automatically URL-encode data
curl -X POST https://api.example.com/search \
  --data-urlencode "query=hello world"
# Result: query=hello%20world

# Encode special characters
curl -X POST https://api.example.com/search \
  --data-urlencode "query=foo&bar=baz"
# Result: query=foo%26bar%3Dbaz

# Encode Chinese characters
curl -X POST https://api.example.com/search \
  --data-urlencode "query=中文搜索"
# Result: query=%E4%B8%AD%E6%96%87%E6%90%9C%E7%B4%A2

2. Manual Encoding

bash
# Use printf for encoding
ENCODED=$(printf '%s' 'hello world' | od -An -tx1 | tr -d ' \n' | sed 's/../%&/g')
curl -X POST -d "query=$ENCODED" https://api.example.com/search

# Use Python for encoding
ENCODED=$(python3 -c "import urllib.parse; print(urllib.parse.quote('hello world'))")
curl -X POST -d "query=$ENCODED" https://api.example.com/search

# Use jq for encoding
ENCODED=$(jq -nr --arg s 'hello world' '$s|@uri')
curl -X POST -d "query=$ENCODED" https://api.example.com/search

Special Characters in URLs

Query Parameters

bash
# Wrong way: & will be interpreted as parameter separator
curl "https://api.example.com/search?q=foo&bar=baz"
# Actually sends two parameters: q=foo and bar=baz

# Correct way: manual encoding
curl "https://api.example.com/search?q=foo%26bar%3Dbaz"

# Or use --data-urlencode (automatically handles &)
curl -G https://api.example.com/search \
  --data-urlencode "q=foo&bar=baz"

Path Special Characters

bash
# Path contains space
curl "https://api.example.com/files/my%20document.pdf"

# Path contains Chinese
curl "https://api.example.com/files/%E4%B8%AD%E6%96%87%E6%96%87%E4%BB%B6.pdf"

# Use --path-as-is to preserve special paths
curl --path-as-is "https://api.example.com/../secret.txt"

Different Scenario Handling

Scenario 1: Form Data Submission

bash
# Use --data-urlencode to auto-encode form data
curl -X POST https://api.example.com/submit \
  --data-urlencode "name=张三" \
  --data-urlencode "email=zhangsan@example.com" \
  --data-urlencode "message=Hello World! 你好世界！"

# Mix encoded and non-encoded data
curl -X POST https://api.example.com/submit \
  -d "id=123" \
  --data-urlencode "content=Special chars: & = ?"

Scenario 2: URL Query Parameters

bash
# Use -G and --data-urlencode to build query string
curl -G https://api.example.com/search \
  --data-urlencode "q=hello world" \
  --data-urlencode "category=技术&编程" \
  --data-urlencode "page=1"

# Result URL:
# https://api.example.com/search?q=hello%20world&category=%E6%8A%80%E6%9C%AF%26%E7%BC%96%E7%A8%8B&page=1

Scenario 3: Special Characters in JSON

bash
# Special characters in JSON need escaping
curl -X POST https://api.example.com/users \
  -H "Content-Type: application/json" \
  -d '{"name":"张三","bio":"Line1\nLine2\tTabbed"}'

# Use jq to handle complex JSON
jq -n '{name: "张三", bio: "Line1\nLine2"}' | \
  curl -X POST https://api.example.com/users \
    -H "Content-Type: application/json" \
    -d @-

Encoding Utility Functions

bash
# URL encoding function
url_encode() {
    local string="$1"
    local encoded=""
    local pos c o
    
    for (( pos=0; pos<${#string}; pos++ )); do
        c=${string:$pos:1}
        case "$c" in
            [-_.~a-zA-Z0-9]) encoded+="$c" ;;
            *) printf -v o '%%%02x' "'$c"; encoded+="$o" ;;
        esac
    done
    echo "$encoded"
}

# Usage example
QUERY=$(url_encode "hello world & more")
curl "https://api.example.com/search?q=$QUERY"

Common Encoding Issues

bash
# Issue 1: Space incorrectly handled
# Wrong:
curl "https://api.example.com/search?q=hello world"
# Correct:
curl "https://api.example.com/search?q=hello%20world"

# Issue 2: & symbol interpreted as separator
# Wrong:
curl "https://api.example.com/search?q=A&B"
# Correct:
curl "https://api.example.com/search?q=A%26B"

# Issue 3: Chinese characters garbled
# Wrong:
curl "https://api.example.com/search?q=中文"
# Correct:
curl "https://api.example.com/search?q=%E4%B8%AD%E6%96%87"

Complete Example Script

bash
#!/bin/bash
# URL encoding handling example

API_BASE="https://api.example.com/v1"

# Encoding function
urlencode() {
    python3 -c "import urllib.parse; print(urllib.parse.quote('''$1'''))"
}

# Search request
search_query="hello world & special chars: 中文"
encoded_query=$(urlencode "$search_query")

echo "Original: $search_query"
echo "Encoded: $encoded_query"

# Send request
curl -G "${API_BASE}/search" \
  --data-urlencode "q=$search_query" \
  -H "Accept: application/json"

# Form submission
form_data="name=张三&email=test@example.com&msg=Hello World!"
curl -X POST "${API_BASE}/submit" \
  --data-urlencode "name=张三" \
  --data-urlencode "email=test@example.com" \
  --data-urlencode "msg=Hello World!"

Best Practices

1. Prefer --data-urlencode: Automatic encoding, avoids errors
2. Use -G for query parameters: Build URLs with --data-urlencode
3. Use tools for JSON: Use jq for complex JSON, automatic escaping
4. Avoid manual URL concatenation: Easy to miss encoding, causes request failures
5. Test encoding results: Use -v to see actual URL sent