VALUES for setting required language in a query

I try to use the VALUES clause to set a language used in different places in a query, e.g.

prefix skos: <http://www.w3.org/2004/02/skos/core#>
select ?s ?prefLabel ?altLabel
where {
?s skos:prefLabel ?prefLabel .
optional {?s skos:altLabel ?altLabel }.
filter (lang(?prefLabel) = ?language && lang(?altLabel) = ?language)

values ?language { "en" }

This gives an empty result. When I move the values clause into the where block, the expected triples are found.

In some cases, the values clause outside the where block works fine (e.g. also the third example at http://www.w3.org/TR/sparql11-query/#inline-data-examples). I'm sure the explanation is in the docs, but I couldn't figure it out.