Commit e820e293 authored by kassings's avatar kassings

Initial commit

parents
*.iml
.idea/
dist/
*.egg-info
*.pyc
program.lp
python-ortools-lp-parser.tar.gz
temp/
.coverage
htmlcov/
!example/program.lp
Copyright 2019 snkas
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
# Python OR-Tools LP Parser
The fact that the or-tools open source project (https://developers.google.com/optimization/) enables a decent linear solver to be included in Python via `pip3` without manual compilation is remarkable. The Python wrapper of the or-tools optimization API (i.e. `pip3 install ortools`) unfortunately does not support the reading of linear program files. This small module is to enable Python developers to read in LP formatted linear program files.
**This fan-written LP parser is NOT IN ANY WAY affiliated with or endorsed by the or-tools developers. This parser is distributed on an "as is" basis, without warranties or conditions of any kind (see also the Apache 2.0 License in ./LICENSE).**
It aims to follow the LPSolve 5.1 LP format (http://lpsolve.sourceforge.net/5.1/lp-format.htm). The format accepted by this parser is different in the following ways:
1. An objective function / constraint / declaration must be on a single line terminated by a semicolon (;)
2. Multi-line comments are not allowed
3. The objective function must have "max:" or "min:" at the start of it
4. Keywords (i.e., "max", "min", "int") must be lowercase
5. Semi-continuous variables are not allowed
6. Special ordered sets (SOS) are not allowed
7. Constraint names are completely ignored (e.g., "my_constraint_row_1: x1 >= x2" is just parsed to "x1 >= x2")
8. At most 1 detached (whitespaced) sign and 1 directly in front of a coefficient/variable is permitted (e.g., "max: --x1 + +3.0" is allowed, "max: --x1 + + +3.0" is not, "max: --x1 + + 3.0" is also not).
## Installation
**Requirements**
* Python 3.5+
* `pip3 install ortools`
**Option 1**
```bash
$ pip3 install git+https://gitlab.inf.ethz.ch/kassings/python-ortools-lp-parser.git
```
You can now include it using: `import ortoolslpparser`
**Option 2**
Clone/download this Git repository. Then, execute the following to install the package locally:
```bash
$ bash install_local.sh
```
You can now include it using: `import ortoolslpparser`
## Getting started
Create a LP formatted linear program called `program.lp`:
```
max: x1 - x2;
x1 >= 0.3;
x1 <= 30.6;
x2 >= 24.9;
x2 <= 50.1;
```
Create a Python file called `example.py`:
```python
import ortoolslpparser
parse_result = ortoolslpparser.parse_lp_file("program.lp")
solver = parse_result["solver"]
solver.Solve()
print("Objective value: %f" % solver.Objective().Value())
for var_name in parse_result["var_names"]:
print("Variable %s: %f" % (var_name, solver.LookupVariable(var_name).solution_value()))
```
Then run `python3 example.py`. It should output:
```
Objective value: 5.700000
Variable x1: 30.600000
Variable x2: 24.900000
```
## Testing
Run all tests:
```bash
$ bash run_tests.sh
```
Calculate coverage (output in htmlcov/):
```bash
$ bash calculate_coverage.sh
```
## General advice
* **Declare the tightest bounds possible.** The default type of a variable is a floating point number in the range (-inf, inf). Make sure that you declare for each variable as tight bounds as possible: this can help the solver. In particular, if you know a certain variable `x` is a non-negative number, add a `x >= 0;` constraint to limit the range to [0, inf). The Glop solver tends to no be as good at solving (-inf, inf)-bounded variables: in some instances it declares the program as ABNORMAL.
#!/usr/bin/env bash
# Copyright 2019 snkas
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
pip3 install coverage
coverage run -m unittest discover -v -s tests
coverage html
rm .coverage
import ortoolslpparser
parse_result = ortoolslpparser.parse_lp_file("program.lp")
solver = parse_result["solver"]
solver.Solve()
print("Objective value: %f" % solver.Objective().Value())
for var_name in parse_result["var_names"]:
print("Variable %s: %f" % (var_name, solver.LookupVariable(var_name).solution_value()))
import ortoolslpparser
parse_result = ortoolslpparser.parse_lp_file("program.lp")
solver = parse_result["solver"]
result = solver.Solve()
if result == solver.OPTIMAL:
print("Value of objective function: %f" % solver.Objective().Value())
print("Actual values of the variables:")
for var_name in parse_result["var_names"]:
print("%s %f" % (var_name, solver.LookupVariable(var_name).solution_value()))
else:
print("Linear program was not solved.")
error_msg = "UNKNOWN"
if result == solver.OPTIMAL:
error_msg = "OPTIMAL"
elif result == solver.FEASIBLE:
error_msg = "FEASIBLE"
elif result == solver.INFEASIBLE:
error_msg = "INFEASIBLE"
elif result == solver.UNBOUNDED:
error_msg = "UNBOUNDED"
elif result == solver.ABNORMAL:
error_msg = "ABNORMAL"
elif result == solver.NOT_SOLVED:
error_msg = "NOT_SOLVED"
print("Error returned by OR-tools: %s (code: %d)" % (error_msg, result))
from ortools.linear_solver import pywraplp
import ortoolslpparser
parse_result = ortoolslpparser.parse_lp_file(
"program_mixed_integer.lp",
use_solver=pywraplp.Solver.CBC_MIXED_INTEGER_PROGRAMMING
)
solver = parse_result["solver"]
solver.Solve()
print("Objective value: %f" % solver.Objective().Value())
for var_name in parse_result["var_names"]:
print("Variable %s: %f" % (var_name, solver.LookupVariable(var_name).solution_value()))
max: x1 - x2;
x1 >= 0.3;
x1 <= 30.6;
x2 >= 24.9;
x2 <= 50.1;
max: x1 - x2;
x1 >= 0.3;
x1 <= 30.6;
x2 >= 24.9;
x2 <= 50.1;
int x1;
#!/usr/bin/env bash
# Copyright 2019 snkas
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
tar -zcf python-ortools-lp-parser.tar.gz ./
pip install python-ortools-lp-parser.tar.gz
rm python-ortools-lp-parser.tar.gz
# Copyright 2019 snkas
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
from .ortoolslpparser import parse_lp_file
# Copyright 2019 snkas
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
from ortools.linear_solver import pywraplp
import re
_REGEXP_SINGLE_VAR_NAME = r"^[A-Za-z]+[A-Za-z0-9_\[\]{}/.&#$%~'@^]*$"
def _is_valid_constant_float(value) -> bool:
if re.search(r"^[+-]?(\d+(\.\d*)?|\.\d+)([eE][-+]?\d+)?$", value.strip()) is None:
return False
else:
return True
def _retrieve_core_line(line: str, line_nr: int) -> str:
# Remove whitespace
line = line.strip()
# Remove any comments
if line.find("//") != -1:
line = line.split("//")[0]
# Remove again any whitespace
line = line.strip()
# Empty lines are allowed
if len(line) == 0:
return line
# Check for the semicolon at the end
if line[-1] != ";":
raise ValueError("Line %d does not end with a semi-colon." % line_nr)
# Remove the semicolon
line = line[0:len(line) - 1]
# Remove again any whitespace
line = line.strip()
# Semicolon-only lines are not allowed
if len(line) == 0:
raise ValueError("Line %d ends with semi-colon but is empty otherwise." % line_nr)
return line
def _parse_declaration(solver: pywraplp.Solver, core_line: str, line_nr: int, var_names: set):
spl_whitespace = core_line.split(maxsplit=1)
if spl_whitespace[0] != "int":
raise ValueError("Declaration on line %d should start with \"int \"." % line_nr)
if len(spl_whitespace) != 2:
raise ValueError("Declaration on line %d has no variables." % line_nr)
spl_variables = spl_whitespace[1].split(",")
for raw_var in spl_variables:
if not re.match(_REGEXP_SINGLE_VAR_NAME, raw_var.strip()):
raise ValueError("Non-permitted variable name (\"%s\") on line %d." % (raw_var, line_nr))
clean_var = raw_var.strip()
var_names.add(clean_var)
solver.IntVar(-solver.infinity(), solver.infinity(), clean_var)
def _set_coefficients(solver: pywraplp.Solver, objective_or_constraint, coefficient_part: str, line_nr: int,
var_names: set):
# Strip the coefficient whitespace
remainder = coefficient_part.strip()
if len(remainder) == 0:
raise ValueError("No variables present in equation on line %d." % line_nr)
running_constant_sum = 0.0
had_at_least_one_variable = False
while len(remainder) != 0:
# Combination sign
coefficient = 1.0
combination_sign_match = re.search(r"^[-+]", remainder)
if not combination_sign_match is None:
if combination_sign_match.group() == "-":
coefficient = -1.0
remainder = remainder[1:].strip()
# Real sign
sign_match = re.search(r"^[-+]", remainder)
if not sign_match is None:
if sign_match.group() == "-":
coefficient = coefficient * -1.0
remainder = remainder[1:]
# Mantissa and exponent
mantissa_exp_match = re.search(r"^(\d+(\.\d*)?|\.\d+)([eE][-+]?\d+)?", remainder)
whitespace_after = False
if not mantissa_exp_match is None:
coefficient = coefficient * float(mantissa_exp_match.group())
remainder = remainder[mantissa_exp_match.span()[1]:]
stripped_remainder = remainder.strip()
if len(remainder) != len(stripped_remainder):
whitespace_after = True
remainder = stripped_remainder
# Variable name
var_name_match = re.search(r"^[A-Za-z]+[A-Za-z0-9_\[\]{}/.&#$%~'@^]*", remainder)
if not var_name_match is None:
# It must have had at least one variable
had_at_least_one_variable = True
# Retrieve clean variable name
clean_var = var_name_match.group()
var_names.add(clean_var)
solver_var = solver.LookupVariable(clean_var)
if solver_var is None:
solver_var = solver.NumVar(-solver.infinity(), solver.infinity(), clean_var)
# Set coefficient
objective_or_constraint.SetCoefficient(solver_var, coefficient)
# Strip what we matched
stripped_remainder = remainder[var_name_match.span()[1]:].strip()
whitespace_after = False
if len(remainder) != len(stripped_remainder):
whitespace_after = True
remainder = stripped_remainder
elif mantissa_exp_match is None:
raise ValueError("Cannot process remainder coefficients of \"%s\" on line %d." % (remainder, line_nr))
else:
running_constant_sum += coefficient
# At the end of each element there must be white space or the next sign, or it was the last one
if len(remainder) != 0 and not whitespace_after and remainder[0:1] != "-" and remainder[0:1] != "+":
raise ValueError("Whitespace or combination sign is missing on line %d." % line_nr)
# There must have been at least one variable
if not had_at_least_one_variable:
raise ValueError("Not a single variable present in the coefficients on line %d." % line_nr)
return running_constant_sum
def _attempt_to_improve_var_bounds_one_hs(solver: pywraplp.Solver, coefficient_part: str, is_leq:bool,
right_hand_side: str):
coefficient_part = coefficient_part.strip()
if re.match(_REGEXP_SINGLE_VAR_NAME, coefficient_part):
if is_leq:
solver.LookupVariable(coefficient_part).SetUb(float(right_hand_side))
else:
solver.LookupVariable(coefficient_part).SetLb(float(right_hand_side))
def _attempt_to_improve_var_bounds_two_hs(solver: pywraplp.Solver, coefficient_part: str, is_leq: bool,
left_hand_side: str, right_hand_side: str):
coefficient_part = coefficient_part.strip()
if re.match(_REGEXP_SINGLE_VAR_NAME, coefficient_part):
if is_leq:
solver.LookupVariable(coefficient_part).SetLb(float(left_hand_side))
solver.LookupVariable(coefficient_part).SetUb(float(right_hand_side))
else:
solver.LookupVariable(coefficient_part).SetLb(float(right_hand_side))
solver.LookupVariable(coefficient_part).SetUb(float(left_hand_side))
def _parse_objective_function(solver: pywraplp.Solver, core_line: str, line_nr: int, var_names: set):
spl_colon = core_line.split(":", maxsplit=1)
objective = solver.Objective()
# Set maximization / minimization if specified
if len(spl_colon) == 1:
raise ValueError("Objective function on line %d must start with \"max:\" or \"min:\"." % line_nr)
else:
if spl_colon[0] != "max" and spl_colon[0] != "min":
raise ValueError("Objective function on line %d must start with \"max:\" or \"min:\"." % line_nr)
elif spl_colon[0] == "max":
objective.SetMaximization()
else:
objective.SetMinimization()
# Set the remainder
coefficient_part = spl_colon[1].strip()
# Finally set the coefficients of the objective
constant = _set_coefficients(solver, objective, coefficient_part, line_nr, var_names)
objective.SetOffset(constant)
def _parse_constraint(solver: pywraplp.Solver, core_line: str, line_nr: int, var_names: set):
# We don't care about the coefficient name before the colon
constraint_part = core_line
spl_colon = core_line.split(":", maxsplit=1)
if len(spl_colon) > 1:
constraint_part = spl_colon[1].strip()
# Equality constraint
if constraint_part.find("=") >= 0 and constraint_part.find("<=") == -1 and constraint_part.find(">=") == -1:
equality_spl = constraint_part.split("=")
if len(equality_spl) > 2:
raise ValueError("Equality constraint on line %d has multiple equal signs." % line_nr)
if not _is_valid_constant_float(equality_spl[1]):
raise ValueError("Right hand side (\"%s\") of equality constraint on line %d is not a float "
"(e.g., variables are not allowed there!)."
% (equality_spl[1], line_nr))
equal_value = float(equality_spl[1])
constraint = solver.Constraint(equal_value, equal_value)
constant = _set_coefficients(solver, constraint, equality_spl[0], line_nr, var_names)
constraint.SetLb(constraint.Lb() - constant)
constraint.SetUb(constraint.Ub() - constant)
_attempt_to_improve_var_bounds_two_hs(solver, equality_spl[0], True, equality_spl[1], equality_spl[1])
# Inequality constraints
else:
# Replace all of these inequality signs, because they are equivalent
constraint_part = constraint_part.replace("<=", "<").replace(">=", ">")
# lower bound < ... < upper bound
if constraint_part.count("<") == 2:
spl = constraint_part.split("<")
if not _is_valid_constant_float(spl[0]):
raise ValueError("Left hand side (\"%s\") of inequality constraint on line %d is not a float "
"(e.g., variables are not allowed there!)."
% (spl[0], line_nr))
if not _is_valid_constant_float(spl[2]):
raise ValueError("Right hand side (\"%s\") of inequality constraint on line %d is not a float "
"(e.g., variables are not allowed there!)."
% (spl[2], line_nr))
constraint = solver.Constraint(float(spl[0]), float(spl[2]))
constant = _set_coefficients(solver, constraint, spl[1], line_nr, var_names)
constraint.SetLb(constraint.Lb() - constant)
constraint.SetUb(constraint.Ub() - constant)
_attempt_to_improve_var_bounds_two_hs(solver, spl[1], True, spl[0], spl[2])
# upper bound > ... > lower bound
elif constraint_part.count(">") == 2:
spl = constraint_part.split(">")
if not _is_valid_constant_float(spl[0]):
raise ValueError("Left hand side (\"%s\") of inequality constraint on line %d is not a float "
"(e.g., variables are not allowed there!)."
% (spl[0], line_nr))
if not _is_valid_constant_float(spl[2]):
raise ValueError("Right hand side (\"%s\") of inequality constraint on line %d is not a float "
"(e.g., variables are not allowed there!)."
% (spl[2], line_nr))
constraint = solver.Constraint(float(spl[2]), float(spl[0]))
constant = _set_coefficients(solver, constraint, spl[1], line_nr, var_names)
constraint.SetLb(constraint.Lb() - constant)
constraint.SetUb(constraint.Ub() - constant)
_attempt_to_improve_var_bounds_two_hs(solver, spl[1], False, spl[0], spl[2])
# ... < upper bound
elif constraint_part.count("<") == 1:
spl = constraint_part.split("<")
if not _is_valid_constant_float(spl[1]):
raise ValueError("Right hand side (\"%s\") of inequality constraint on line %d is not a float "
"(e.g., variables are not allowed there!)."
% (spl[1], line_nr))
constraint = solver.Constraint(-solver.infinity(), float(spl[1]))
constant = _set_coefficients(solver, constraint, spl[0], line_nr, var_names)
constraint.SetUb(constraint.Ub() - constant)
_attempt_to_improve_var_bounds_one_hs(solver, spl[0], True, spl[1])
# ... > lower bound
elif constraint_part.count(">") == 1:
spl = constraint_part.split(">")
if not _is_valid_constant_float(spl[1]):
raise ValueError("Right hand side (\"%s\") of inequality constraint on line %d is not a float "
"(e.g., variables are not allowed there!)."
% (spl[1], line_nr))
constraint = solver.Constraint(float(spl[1]), solver.infinity())
constant = _set_coefficients(solver, constraint, spl[0], line_nr, var_names)
constraint.SetLb(constraint.Lb() - constant)
_attempt_to_improve_var_bounds_one_hs(solver, spl[0], False, spl[1])
# ...
elif constraint_part.count(">") == 0 and constraint_part.count("<") == 0:
raise ValueError("No (in)equality sign present for constraint on line %d." % line_nr)
# Some strange combination
else:
raise ValueError("Too many (in)equality signs present for constraint on line %d." % line_nr)
def _set_declarations(solver: pywraplp.Solver, lp_filename: str, var_names: set):
with open(lp_filename, "r") as lp_file:
line_nr = 0
in_objective_function = True
in_constraints = False
for line in lp_file:
line_nr += 1
# Retrieve the core of the line without heading or trailing whitespace and without comments
core_line = _retrieve_core_line(line, line_nr)
# Skip over empty lines
if len(core_line) == 0:
continue
# The first non-empty core line must be the objective function
if in_objective_function:
in_objective_function = False
in_constraints = True
# Go over until we hit a line without colon and starting with "int<whitespace>"
elif in_constraints:
if core_line.find(":") == -1 and core_line.split()[0] == "int":
in_constraints = False
_parse_declaration(solver, core_line, line_nr, var_names)
# Any core line from here on must be a declaration
else:
_parse_declaration(solver, core_line, line_nr, var_names)
def _set_objective_function_and_constraints(solver: pywraplp.Solver, lp_filename: str, var_names: set):
with open(lp_filename, "r") as lp_file:
line_nr = 0
in_objective_function = True
in_constraints = False
for line in lp_file:
line_nr += 1
# Retrieve the core of the line without heading or trailing whitespace and without comments
core_line = _retrieve_core_line(line, line_nr)
# Skip over empty lines
if len(core_line) == 0:
continue
# The first non-empty core line must be the objective function
if in_objective_function:
_parse_objective_function(solver, core_line, line_nr, var_names)
in_objective_function = False
in_constraints = True
# Go over until we hit a line without colon and starting with "int<whitespace>"
elif in_constraints:
if core_line.find(":") == -1 and core_line.split()[0] == "int":
break
else:
_parse_constraint(solver, core_line, line_nr, var_names)
def parse_lp_file(lp_filename: str, use_solver=pywraplp.Solver.GLOP_LINEAR_PROGRAMMING):
"""
Read in a linear program defined in a LP file.
The LP specification format is similar to LPSolve 5.1.
For the accepted format, consult:
https://github.com/snkas/python-ortools-lp-parser
:param lp_filename: .lp file name (i.e. "/path/to/file.lp")
:param use_solver: Solver instance to use (optional)
(1) pywraplp.Solver.CLP_LINEAR_PROGRAMMING)
(2) pywraplp.Solver.GLOP_LINEAR_PROGRAMMING) -> Default
(3) pywraplp.Solver.CBC_MIXED_INTEGER_PROGRAMMING)
(4) pywraplp.Solver.BOP_INTEGER_PROGRAMMING)
:return: Dictionary {
"solver": pywraplp.Solver instance with the linear program in it
"var_names": set of all variable names in the solver
}
"""
# Set of the names of all variables
var_names = set()
# Solver instantiation