NEW YORK (Reuters) - A Washington-based law firm co-founded by former FBI director Louis J. Freeh has attested that as of June 1 cryptocurrency firm Tether had enough U.S. dollar reserves to back its virtual coins in circulation, according to a report released by Tether on Wednesday.

The report by Freeh, Sporkin & Sullivan LLP, comes a week after a University of Texas research paper alleged Tether’s token could have been used to manipulate bitcoin’s price last year during its meteoric rise.

Tether critics have raised concerns over the past year about whether it actually holds $1 dollar in reserve for every token called tether issued, as it claimed.

The FSS report is not a full audit of Tether, but states the law firm received sworn and notarized statements from Tether’s two banks which indicated the company had approximately $2.55 billion in its accounts on June 1.

“FSS is confident that Tether’s unencumbered assets exceed the balance of fully-backed USD Tethers in circulation as of June 1st, 2018,” the report said.

The day of the examination was selected by FSS, the law firm said in the report, adding that it conducted “comprehensive examinations and telephone interviews of key personnel at Tether and its banks.”

The report did not disclose the names of the banks and also stated “FSS did not, as part of the Engagement, arrive at any conclusions as to Tether’s compliance with applicable laws and regulations in any jurisdiction.”

Eugene Sullivan, a former federal judge and FSS cofounder, is an advisory board member of one of Tether’s banks, the report said.

FSS did not immediately respond to a request for comment.

Tether shares investors and management with Bitfinex, one of the world’s largest cryptocurrency exchanges.

“These allegations of manipulation are ... just completely misplaced,” Stuart Hoegner, Bitfinex’s general counsel, said in an interview.

Tether hired FSS in March and is in discussions with auditors for a full audit, Hoegner said.

Global regulators have been scrutinizing cryptocurrency markets following a rally in prices last year. The U.S. Commodity Futures Trading Commission and the Department of Justice are reportedly investigating whether the price of bitcoin is being manipulated.

In December the CFTC sent a subpoena to Tether and Bitfinex.

Hoegner declined to comment on whether Tether or Bitfinex were under investigation by the CFTC or the DOJ. He also declined to share the contents of the subpoena.

One of the main selling points for Python is that it is dynamically-typed. There is no plan to change this. Nevertheless, in September 2014 Guido van Rossum (Python BDFL) created a python enhancement proposal (PEP-484) to add type hints to Python. It has been released for general usage a year later, in September 2015, as part of Python 3.5.0. Twenty-five years into its existence now there was a standard way to add type information to Python code. In this blog post, I'll explore how the system matured, how you can use it and what's next for type hints.

Disclaimer: throughout this blog post, you'll see many seals and penguin pictures, the reason for this is mostly my admiration for these animals; and hey nothing like some cute animals to help digest some complex topics, not?

What it was designed to do?

First, let's see why do we need type hints in Python. There are multiple advantages of this, and I'll try to enumerate it in their order of importance:

1. Easier to reason about code

Knowing the type of the parameters makes it a lot easier to understand and maintain a code base. For example, let's assume you have a function. While we do know the parameters types at the time of creating the function, a few months down the line this is no longer the case. Having stated the types of all parameters and return types right beside the code can speed up significantly the time required to catch up with a code snippet. Always remember that code you read code a lot more often than you write it. Therefore you should optimise for ease of reading.

Having type hints informs you of what parameter types you need to pass on when calling a function, and when you need to extend/modify the function tells you about the type of data you get both as input and output. For example, imagine the following send request function,

def send_request(request_data : Any,
                 headers: Optional[Dict[str, str]],
                 user_id: Optional[UserId] = None,
                 as_json: bool = True):
    ...

Just looking at the signature of this I know that while the request_data could be anything, the headers content is a dictionary of strings. The user information is optional (defaulting to None) or it needs to be whatever UserId encodes it too. Also the contract for as_json is that it needs to be always a boolean value, being a flag essentially even though the name might not suggest that at first.

The truth is many of us already understand that type information is essential, however in lack of better options until now this was often mentioned inside the docstring. The type hint system moves this closer to the interface of the function and provides a well-defined way to declare complex type requirements. Building linters that can check these type hint constraints ensure that they never become out of date, granted that you run them after every code change.

2.Easier refactoring

Type hints make it trivial to find where a given class is used when you're trying to refactor your code base. While many IDEs already have some heuristic in place to achieve this, type hints allow them to have 100% detection and accuracy ratio. Generally offers a smoother and more accurate detection of how types run through your code. Remember while dynamic typing means any variable can become any of types, all your variables have at all time one and only one type. Type system still is very much a core component of programming, remember all the time you've used isinstance to drive your application logic.

3. Easier to use libraries

Having type hints mean IDEs can have a more accurate and smarter suggestion engine. Now when you invoke auto-complete, the IDE knows with complete confidence what methods/attributes are available on an object. Furthermore, if the user tries to call something which is non-existent or passes arguments of an incorrect type the IDE can instantly warn about it.

4. Type linters

While the IDE suggesting incorrect argument types is nice, an extension of this is to have a linter tool that makes sure that type wise the logic of your application is sound. Running this tool can help you catch bugs early on (e.g. in the example that follows the input must be of type str, passing in None throws an exception):

def transform(arg):
    return 'transformed value {}'.format(arg.upper())
transform(None) # if arg would be type hinted as str the type linter could warn that this is an invalid call

While in this trivial case, some could argue that it's easy to see the mismatch, remember this works in more complicated cases too, where such mismatches get harder and harder to see; such as nested function calls:

def construct(param=None):
    return None if param is None else ''

def append(arg):
    return arg + ' appended'
    
transform( append( construct() ) )

While there are more and more linters out there, the reference implementation of the Python type checking is mypy. mypy is a Python command line application, making it easy to integrate into a continuous integration pipeline.

5. Runtime data validation

Type hints can be used to validate at runtime to ensure that the caller does not break the contract of methods. It is no longer needed to start your function with a long list of type asserts; instead, use a framework that re-uses type hints and automatically checks that they are meet before your business logic runs (for example with pydantic):

from datetime import datetime
from typing import List
from pydantic import BaseModel, ValidationError

class User(BaseModel):
    id: int
    name = 'John Doe'
    signup_ts: datetime = None
    friends: List[int] = []

external_data = {'id': '123', 'signup_ts': '2017-06-01 12:22',
                 'friends': [1, 2, 3]}
user = User(**external_data)

try:
    User(signup_ts='broken', friends=[1, 2, 'not number'])
except ValidationError as e:
    print(e.json())

What it wasn't designed to do?

From the get-go, Guido clearly stated that type hints are not meant to be used for the following use cases (of course that does not mean that people do not have libraries/tools outside which do just that - open source power for the win!):

1. No runtime type inference

The runtime interpreter (CPython) does not try to deduce type information at runtime, and perhaps validate arguments passed around based on that.

2. No performance tuning

The runtime interpreter (CPython) does not use the type information to optimise the generated bytecode for either security or performance. When executing a Python script type hints are treated just like comments; the interpreter discards it.

The key takeaway should be that type hints are designed to improve developer experience, not to influence how your script evaluates. It creates happy developers, not faster code!

Python has gradual type hinting; meaning that whenever for a given function or variable the type hint is not specified we assume that it can have any type (that is it remains a dynamically typed section). Use this to gradually make your code base type-aware, one function or variable at a time. It is possible to type hint:

• function arguments,
• function return values,
• variables.

Remember only type hinted code are type checked! When you run the linter (e.g. mypy) on a type hinted code you'll get errors if there are type miss-matches:

# tests/test_magic_field.py
f = MagicField(name=1, MagicType.DEFAULT)
f.names()

This code will generate the following output:

[email protected] ~/python-magic (master●)$ mypy --ignore-missing-imports tests/test_magic_field.py
tests/test_magic_field.py:21: error: Argument 1 to "MagicField" has incompatible type "int";
    expected "Union[str, bytes]"
tests/test_magic_field.py:22: error: "MagicField" has no attribute "names"; maybe "name" or "_name"?

Note we can detect both type incompatibility for the argument passed in, and accesses to inexistent attributes on objects. The later even suggests valid options available, making it easy to notice and fix typos.

How to add it

Once you decide to add type hints, you'll come to realise that you can add it in more than one ways to the code base. Let's see what your options are.

1. Type annotations

from typing import List

class A(object):
    def __init__() -> None:
         self.elements : List[int] = []

   def add(element: int) -> None:
         self.elements.append(element)

Type annotations is the straightforward way and is the one you'll find mostly mentioned on the typing documentation. It uses function annotations added to language via PEP-3107 (Python 3.0+) and variable annotations via PEP-526 (Python 3.6+). These allow you to use the : syntax to attach information to variables and function arguments. The -> operator is used to attach information to the return value of a function/method.

The upside of this method is that:

The downside of it is that:

The last two point contradicts the initial goal of the type system we enlisted before: that is to handle all type information basically as a comment during runtime. To resolve some of this contradiction Python 3.7 introduces PEP-563 ~ postponed evaluation of annotations. Once you add the import of:

from __future__ import annotations

The interpreter will no longer construct these compound types. Once the interpreter parses the scripts syntax tree, it identifies type hints and bypasses evaluating it, keeping it as raw strings. This mechanism allows for type hint interpretation to happen where they need to: by the linter when it runs type checks. Once the mythical Python 4 comes to life, this mechanism shall be the default behaviour.

When the annotation syntax is not available one can use the type comments:

from typing import List

class A(object):
    def __init__():
         # type: () -> None
         self.elements = []  # type: List[int]

   def add(element):
         # type: (List[int]) -> None
         self.elements.append(element)

Going down this path, we do get some benefits:

But we also generate some new problems:

To avoid having long lines of code as type hint, it's possible to type hint arguments one by one via type comments, and then put in the line after only the return type annotation:

def add(element # type: List[int]
       ):
    # type: (...) -> None
    self.elements.append(element)

Let's have a quick use look at how type comments can make your code messier. Below is a code snippet that swaps out two properties values inside a class. Fairly trivial:

@contextmanager
def swap_in_state(state, config, overrides):
    old_config, old_overrides = state.config, state.overrides
    state.config, state.overrides = config, overrides
    yield old_config, old_overrides
    state.config, state.overrides = old_config, old_overrides

First, you must add type hints. Because the type hint would be long winded you attach type hint argument by argument:

@contextmanager
def swap_in_state(state,  # type: State
                  config,  # type: HasGetSetMutable
                  overrides  # type: Optional[HasGetSetMutable]
                 ):
# type: (...) -> Generator[Tuple[HasGetSetMutable, Optional[HasGetSetMutable]], None, None]
    old_config, old_overrides = state.config, state.overrides
    state.config, state.overrides = config, overrides
    yield old_config, old_overrides
    state.config, state.overrides = old_config, old_overrides

However, wait you need to import your types used:

from typing import Generator, Tuple, Optional
from magic import RunSate

HasGetSetMutable = Union[Dict, List]

@contextmanager
def swap_in_state(state,  # type: State
                  config,  # type: HasGetSetMutable
                  overrides  # type: Optional[HasGetSetMutable]
                  ):
    # type: (...) -> Generator[Tuple[HasGetSetMutable, Optional[HasGetSetMutable]], None, None]
    old_config, old_overrides = state.config, state.overrides
    state.config, state.overrides = config, overrides
    yield old_config, old_overrides
    state.config, state.overrides = old_config, old_overrides

Now formatting like this the code causes some false positives in the static linter (e.g. pylint here) so you add a few suppress comments for this:

from typing import Generator, Tuple, Optional, Dict, List
from magic import RunSate

HasGetSetMutable = Union[Dict, List]  # pylint: disable=invalid-name

@contextmanager
def swap_in_state(state,  # type: State
                   config,  # type: HasGetSetMutable
                   overrides  # type: Optional[HasGetSetMutable]
                   ):  # pylint: disable=bad-continuation
    # type: (...) -> Generator[Tuple[HasGetSetMutable, Optional[HasGetSetMutable]], None, None]
    old_config, old_overrides = state.config, state.overrides
    state.config, state.overrides = config, overrides
    yield old_config, old_overrides
    state.config, state.overrides = old_config, old_overrides

Now you're done. Nevertheless, you made your six lines of code sixteen lines long. Yay, more code to maintain! Increasing your code base only sounds good if you're getting paid by the number line of code written and your manager is complaining you're not performing well enough.

3. Interface stub files

This option allows you to keep your code as it is:

class A(object):
  def __init__() -> None:
      self.elements = []

  def add(element):
      self.elements.append(element)

and instead add another file with pyi extension right beside it:

# a.pyi alongside a.py
from typing import List

class A(object):
  elements = ... # type: List[int]
  def __init__() -> None: ...
  def add(element: int) -> None: ...

Interface files are not a new thing, C/C++ had it for decades now. Because Python is an interpreted language it does not need it usually, however as every problem in computer science can be solved by adding a new level of indirection, we can add it to store the type information.

The upside of this is that:

Now there are also some hefty penalties to pay:

The last two drawback makes it especially hard to check that the type hinted code base via a stub file is in sync or not. In this current form type stubs are a way to provide type hints to your users, but not for yourself, and are especially hard to maintain. To fix these, I've taken up the task of merging stub files with source files inside mypy; in theory, would fix both problems - you can follow on its progress under python/mypy ~ issue 5208.

4. Docstrings

It is possible to add type information into docstrings too. Even though this is not part of the type hint framework designed for Python, it is supported by most mainstream IDEs. They are mostly the legacy way of doing this.

On the plus side:

However, it has serious flaws in the form of:

What to add?

Let's dive into the specifics though. For an exhaustive list of what type information you can add, please see the official documentation. Here I'll do a quick 3-minute overview for you to get the idea of it. There are two types of type categories: nominal types and duck types (protocols).

1. Nominal type

Nominal types are types that have a name to it within the Python interpreter. For example all builtin types (int, bolean, float, type, object etc). Then we have the generic types which mostly manifest in form of the containers:

t : Tuple[int, float] = 0, 1.2
d : Dict[str, int] = {"a": 1, "b": 2}
d : MutableMapping[str, int] = {"a": 1, "b": 2}
l : List[int] = [1, 2, 3]
i : Iterable[Text] = [ u'1', u'2', u'3']

For compound types, it can become cumbersome to keep writing it again and again, so the system allows you to alias types, via:

OptFList = Optional[List[float]]

One can even elevate builtin types to represent their own type, which can be useful to avoid errors where for example you pass in two arguments with the same type in the wrong order to a function:

UserId = NewType('UserId', int)
user_id = UserId(524313)
count = 1
call_with_user_id_n_times(user_id, count)

For namedtuple you can attach your type information directly (note the strong resemblance to a data class from Python 3.7+ or the great attrs library):

class Employee(NamedTuple):
     name: str
     id: int

You have the composing types of one of and optional of:

Union[None, int, str] # one of
Optional[float] # either None or float

You can even type hint your callback functions:

# syntax is Callable[[Arg1Type, Arg2Type], ReturnType]
def feeder(get_next_item: Callable[[], str]) -> None:

One can define it's own generic containers by using the TypeVar construction:

T = TypeVar('T')
class Magic(Generic[T]):
      def __init__(self, value: T) -> None:
         self.value : T = value

 def square_values(vars: Iterable[Magic[int]]) -> None:
     v.value = v.value * v.value

Finally, disable type checking wherever it's not needed by using the Any type hint:

def foo(item: Any) -> int:
     item.bar()

2. Duck types - protocols

In this case instead of having an actual type one can be more Pythonic, and go with the theorem that if it quacks like a duck, and acts like a duck, then most definitely for all intended purposes it is a duck. In this case, you define what operations and attributes you expect on objects instead of explicitly stating their types. The grounds of this were laid down in PEP-544 ~ Protocols.

KEY = TypeVar('KEY', contravariant=true)

# this is a protocol having a generic type as argument
# it has a class variable of type var, and a getter with the same key type
class MagicGetter(Protocol[KEY], Sized):
    var : KEY
    def __getitem__(self, item: KEY) -> int: ...

def func_int(param: MagicGetter[int]) -> int:
    return param['a'] * 2

def func_str(param: MagicGetter[str]) -> str:
    return '{}'.format(param['a'])

Once you start adding type hints to a code base watch out that sometimes you may experience some oddities. During this moments you might have the what the hell* expression of the following seal:

In this section, I'll try to present a few of these to give you a heads up on what kind of oddities you may run into while adding type information to your code base.

1. str difference in between Python 2/3

Here's a quick implementation of the repr dunder method for a class:

from __future__ import unicode_literals

class A(object):
    def __repr__(self) -> str:
        return 'A({})'.format(self.full_name)

This code has a bug in it. While this is correct under Python 3, it is not under Python 3 (because Python 2 expects to return bytes from repr, however, the unicode_literals import makes the returned value of type unicode). Having the from future import in place means it's not possible to write a repr that satisfies the type requirements for both Python 2 and 3. You need to add runtime logic to do the right thing:

from __future__ import unicode_literals

class A(object):
    def __repr__(self) -> str:
        res = 'A({})'.format(self.full_name)
        if sys.version_info > (3, 0):
            # noinspection PyTypeChecker
            return res
        # noinspection PyTypeChecker
        return res.encode('utf-8')

Now to fight the IDE to accept this form you need to add a few linter comments, which makes this code ever so complicated to read. More importantly, now you have an extra runtime check forced to your type checker.

2. Multiple return types

Imagine you want to write a function that multiplies either a string or an int by two. The first take on this would be:

def magic(i: Union[str, int]) -> Union[str, int]:
    return i * 2

Your input is either str or int, and your return value accordingly is also either str or int. However, if you do it like so, you're telling to the type hint that it really can be either of for both types of inputs. Therefore on the call side, you need to assert the type your calling with:

def other_func() -> int:
    result = magic(2)
    assert isinstance(result, int)
    return result

This inconvenience may determine some people to avoid the call side hassle by making the return value Any. However, there's a better solution. The type hint system allows you to define overloads. Overloads express that for a given input type only a given output type is returned. So in this case:

from typing import overload

@overload
def magic(i: int) -> int:
    pass

@overload
def magic(i: str) -> str:
    pass

def magic(i: Union[int, str]) -> Union[int, str]:
    return i * 2

def other_func() -> int:
    result = magic(2)
    return result

There is a downside to this though. Now your static linter tool is complaining that you're redefining functions with the same name; this is a false positive so add the static linter disable comment mark (# pylint: disable=function-redefined ).

3. Type lookup

Imagine you have a class that allows representing the contained data as multiple types, or that has fields of the different type. You want the user to have a quick and easy way to refer to them, so you add a function, having a built-in types name:

class A(object):
    def float(self):
            # type: () -> float
           return 1.0

Once you run the linter you'll see:

test.py:3: error: Invalid type "test.A.float"

One might ask at this point, what the hell? I've defined the return value as float not as test.A.float. The reason for this obscure error is that the type hinter resolves types by evaluating each scope outbound from the definition location. Once it finds a name match, it stops. The first level where it looks is within class A where it finds a float (a function that is) and substitutes that float in.

Now the solution to not run into this issue is to explicitly define that we don't just want any float, but that we want the builtin.float:

if typing.TYPE_CHECKING:
    import builtins

class A(object):
    def float(self):
            # type: () -> builtins.float
           return 1.0

Note that to do this you also need to import builtins, and to avoid this causing issues at runtime, you can guard it with the typing.TYPE_CHECKING flag which is true only during the type linter evaluation, always false otherwise.

4. Contravariant argument

Examine the following use case. You define an abstract base class that contains common operations. Then you have specific classes which handle one type and one type only. You control the creation of the classes which ensures the correct type is passed, and the base is abstract, so this seems an agreeable design:

from abc import ABCMeta, abstractmethod
from typing import Union

class A(metaclass=ABCMeta):
    @abstractmethod
    def func(self, key):  # type: (Union[int, str]) -> str
        raise NotImplementedError

class B(A):
    def func(self, key):  # type: (int) -> str
        return str(key)

class C(A):
    def func(self, key):  # type: (str) -> str
        return key

However, once you run a type linter check on this you'll find:

test.py:12: error: Argument 1 of "func" incompatible with supertype "A"
test.py:17: error: Argument 1 of "func" incompatible with supertype "A"

The reason for this is that arguments to classes are contravariant. This translates in on scientific terms in your derived class you must handle all types from your parents. However, you may add additional types too. That is even in the function arguments you can only extend what you cover, but not to constrain it in any way:

from abc import ABCMeta, abstractmethod
from typing import Union

class A(metaclass=ABCMeta):
    @abstractmethod
    def func(self, key):  # type: (Union[int, str]) -> str
        raise NotImplementedError

class B(A):
    def func(self, key):  # type: (Union[int, str, bool]) -> str
        return str(key)

class C(A):
    def func(self, key):  # type: (Union[int, str, List]) -> str
        return key

5. Compatibility

See if you can spot the error in the following code snippet:

class A:
    @classmethod
    def magic(cls, a: int) -> 'A':
        return cls()

class B(A):
    @classmethod
    def magic(cls, a: int, b: bool) -> 'B':
        return cls()

If you did not manage yet consider what will happen if you write the following script:

from typing import List, Type

elements : List[Type[A]] = [A, B]
print( [e.magic(1) for e in elements])

Were you to try to run it this would fail with the following runtime error:

    print( [e.magic(1) for e in elements])
TypeError: magic() missing 1 required positional argument: 'b'

The reason being that B is a subtype of A. Therefore it can go into a container of A types (because it extends it, so it can do more than A). However the class method definition for B breaks this contract, it can no longer call magic with just one argument. Moreover, the type linter would fail to point out just this:

test.py:9: error: Signature of "magic" incompatible with supertype "A"

A quick and easy fix for this is to make sure B.magic does work with one argument by making the second optional for example. Now take what we learned to take a look at the following:

class A:
    def __init__(self, a: int) -> None:
        pass

class B(A):
    def __init__(self, a: int, b: bool) -> None:
        super().__init__(a)

What do you think will happen here? Note we moved class methods into constructors, and made no other change, so our script also needs just a slight modification:

from typing import List, Type

elements : List[Type[A]]= [A, B]
print( [e(1) for e in elements])

Here's the runtime error, being mostly the same, just now complaining about __init__ instead of magic:

    print( [e(1) for e in elements])
TypeError: __init__() missing 1 required positional argument: 'b'

So what do you think mypy will say? Were you to run it you'll find that mypy chooses to stay silent. Yes, it will mark this as correct, even though at runtime it fails. The mypy creators said that they found too common of type miss-match to prohibit incompatible __init__ and __new__.

When you hit the wall

So, in conclusion, watch out, type hints sometimes cause strange warnings, which brings out the following feelings summarised in a tweet:

Remember you have some tools at hand that help you discover, understand and perhaps handle these edge cases:

• use reveal_type to see inferred type

  a = [4]
  reveal_type(a)         # -> error: Revealed type is 'builtins.list[builtins.int*]'

• use cast to force a given type:

  from typing import List, cast
  a = [4]
  b = cast(List[int], a) # passes fine
  c = cast(List[str], a) # type: List[str] # passes fine (no runtime check)
  reveal_type(c)         # -> error: Revealed type is 'builtins.list[builtins.str]'

• use the type ignore marker to disable an error in a line:

  x = confusing_function() # type: ignore # see mypy/issues/1167

• ask the community; expose a minimal reproducible version of the problem under the python/typing Gitter chat.

Here's a non exhaustive list of tools built around the type hint system.

type checkers

Use these tools to check against type safety inside your library or application:

1. mypy - Python (the reference type linting tool)
2. pyre - Facebook - Python 3 only, but faster than mypy. An interesting use case of this is the ability to do taint/security code analysis with it - see Pieter Hooimeijer - Types, Deeper Static Analysis, and you.
3. pytype - Google.

type annotation generators

When you want to add type annotations to an existing code base use these to automate the boring part:

1. mypy stubgen command line (see)
2. pyannotate - Dropbox - use your tests to generate type information.
3. monkeytype - Instagram. Fun fact: Instagram actualy uses to run it in their production system: it's triggered once for every million cal (makes the code run five times slower, but once every million calls makes it not that noticeable).

runtime code evaluator

Use these tools to check at runtime if the input arguments to your function/method are of correct type or not:

1. pydantic
2. enforce
3. pytypes

Documentation enrichment - merge docstrings and type hints

In the first part of this blog article, we mentioned that historically people have already stored type information inside docstrings. The reason for this is that your type data is part of your contract, you do want to have type information for your library inside the documentation. So the question remains given that you did not choose to use docstrings as the primary type information storage system, how can you still have them in the docstrings for your documentation?

The answer varies depending on the tool you're using to generate that documentation. However, I'm going to present here an option by using the most popular tool and format in Python: Sphinx and HTML.

Having type information explicitly stated in both the type hints and the docstring is the sure way of eventually having conflicts in between them. You can count on someone at some point is going to update in one place but not in the other. Therefore let's strip all type data from the docstring and have it only as type hints. Now, all we need to do is at documentation build time fetch it from the type hints and insert it into the documentation.

In Sphinx, you can achieve this by having a plugin. The most popular already made version of this is agronholm/sphinx-autodoc-typehints. This tool does two things:

• first, for each function/variable to be documented, it fetches the type hint information;
• then, it transforms the Python types into a docstring representation (this involves recursively unwrapping all the nested type classes, and replacing the type with its string representation);
• finally, appending to the correct parameter into the docstring.

For example Any maps to py:data:`~typing.Any`. Things can get even more complicated for compound types such as Mapping[str, bool] needs to be translated for example too :class:`~typing.Mapping`\\[:class:`str`, :class:`bool`]. Getting the translation here right (e.g. having class or data namespace) is essential so that the intersphinx plugin will work correctly (a plugin that links types directly to their respective Python standard library documentation link).

In order to use it one needs to install it via pip install sphinx-autodoc-types>=2.1.1 and then enable in inside the conf.py file:

# conf.py
extensions = ['sphinx_autodoc_typehints']

That's it all. An example use case of this is RookieGameDevs/revived documentation. For example, given the following source code:

You can get the following output:

So at the end of this long blog post, you may ask: is it worth using type hints, or when should one use them? I think type hinting is at the end of the day essentially the same as your unit tests, just expressed differently in code. What they provide is a standard (and re-usable for other goals) way, to test the input and output types of your codebase.

Therefore, type hints should be used whenever unit test are worth writing. This can be even just ten lines of code if you need to maintain it later. Similarly, you should start adding type hints whenever you start writing unit tests. The only place when I would not add them is when I don't write unit tests, such REPL lines, or throw away one-time usage scripts.

Remember that, similar to unit tests, while it does makes your code base contain an extra number of lines, at the end of the day all the code you add is code that is automatically checked and enforced to be correct. It acts as a safety net to ensure that when you change things around later on things keep working, so probably worth paying this extra cost.

Grace Hopper? Cardinal Barberini? Earl of Peterborough? David Hernandez? Helen Pajama? St. Benedict? Anonymous?

Dear Quote Investigator: People who are eager to initiate a task often cite the following guidance. Here are two versions:

• It’s easier to ask forgiveness than to get permission.
• It’s easier to apologize than to get permission.

This notion has been credited to Grace Murray Hopper who was a U.S. Navy Rear Admiral and pioneering computer scientist. Would you please explore this saying?

Quote Investigator: Grace Hopper did employ and help to popularize the expression by 1982, but it was already in circulation.

The earliest match located by QI appeared in 1846 within a multivolume work called “Lives of the Queens of England” by Agnes Strickland. The ninth volume discussed marriage advice offered by a powerful church official. Emphasis added to excerpts by QI: ¹

But, in truth, the cardinal Barberini … did frankly advise the duchess of Modena to conclude the marriage at once; it being less difficult to obtain forgiveness for it after it was done, than permission for doing it.

A footnote listed the source of the passage above as “Earl of Peterborough, in the Mordaunt Genealogies”. Strictly speaking, the statement was not presented as a proverb; instead, it was guidance tailored to one particular circumstance.

In 1894 a newspaper in Pittsburgh, Pennsylvania printed a thematically related adage within a story about mischievous children: ²

The boys, let me add, every one had respectable parents and who would not, for an instant, have allowed such a prank had they known of its existence; but it is easier to beg forgiveness after the deed is performed.

Another match occurred in the 1903 novel “A Professional Rider” by Mrs. Edward Kennard, but the form was not proverbial: ³

Once married, it would be infinitely easier to ask her father’s forgiveness, than to beg his permission beforehand.

In 1966 “Southern Education Report” printed an instance of the proverb spoken by David Hernandez who was a project director working for the U.S. government program Head Start: ⁴

Hernandez began advertising for bids on the mobile classrooms even before the money to pay for them had been approved. ‘It’s easier to get forgiveness than permission,’ he explained.

The above citation appeared in “The Dictionary of Modern Proverbs” from Yale University Press.

Below are additional selected citations in chronological order.

In 1970 “A Door Will Open” by Helen Pajama printed the adage. The book presented the story of big band trumpet player Randy Brooks who spoke extensively to Pajama: ⁵

Sometimes it’s easier to be forgiven than to get permission. So, we made the move and then notified those concerned where he would be. There wasn’t much criticism to my knowledge.

In February 1971 “The Reader’s Digest” printed an anecdote in a section called “Humor in Uniform” that included a remark attributed to an anonymous military man: ⁶

. . . “Soldier, don’t you know you are not supposed to be using this vehicle for such a purpose?”

Taking a nervous gulp, the young GI replied, “Yes, sir. But sometimes forgiveness is easier to obtain than permission.”

Capt. David L. Benton III (Fort Sill, Okla.)

In September 1971 a newspaper in Rochester, New York reported the observation of an unhappy government planner: ⁷

Town Planning Board Chairman Harry Ewens, a guest at yesterday’s luncheon, echoed Strong’s sentiments and complained that he’s found people unwilling to obtain building permits before beginning construction.

“They’ve found it a lot easier to get forgiveness than to get permission,” Ewens said.

In 1978 the adage was attributed to St. Benedict by an environmental activist who was quoted in a St. Louis, Missouri newspaper: ⁸

David D. Comey, head of Citizens for a Better Environment, a Chicago-based environmental group, said the process illustrates the wisdom of St. Benedict.

It was Benedict who observed more than 1400 years ago that “It is easier to beg forgiveness than to seek permission.”

In 1980 “Murphy’s Law Book Two” compiled by Arthur Bloch contained the following entry: ⁹

STEWART’S LAW OF RETROACTION:
It is easier to get forgiveness than permission.

In 1982 the “Chicago Tribune” of Illinois reported on a speech delivered by Grace Hopper at Lake Forest College which included the following: ¹⁰

“Always remember that it’s much easier to apologize than to get permission,” she said. “In this world of computers, the best thing to do is to do it.”

In conclusion, this article presents a snapshot of current research. The proverbial form was in use by 1966 as shown by the statement from David Hernandez. QI hypothesizes that it was circulating before that date and was anonymous.

Image Notes: Stamps for approved and declined from TayebMEZAHDIA at Pixabay.

(Great thanks to Claudia, William Flesch, and Nick Rollins whose inquiries led QI to formulate this question and perform this exploration. Some citations were previously shared by QI in 2010 on the Freakonomics website. Thanks to Barry Popik, Bill Mullins, Fred Shapiro, and Herb Stahlke for their research and comments. Further thanks to Stephen Goranson for finding the 1903 citation and for verifying the February 1971 citation.)

The office-space empire WeWork was founded eight years ago in New York. It currently leases 240,000 sq metres of real estate in London alone, which reportedly makes it the city’s largest user of offices after the British government. The basic deal is simple enough: you can either pay to put your laptop wherever there is space, or stump up a little more for a more dependable desk or entire office – and, in either case, take advantage of the fact that, with operations in 20 countries, WeWork offers the chance to traverse the planet and temporarily set up shop in no end of locations.

Part of the WeWork idea, moreover, is that a place to toil is only part of what is on offer. As well as your workspace, there will be free beer on tap, regular yoga and pilates sessions, and more. As the working day winds on and such distractions – along with the necessity of meeting other footloose hotshots, and comparing “projects” – take up more of your time, a couple of questions might spring to mind: what is work, and what is leisure? And does the distinction even count for much any more?

Other customers of the company may be troubled by an even more fundamental conundrum: where is their workplace – and what, by contrast, constitutes home? WeWork is slowly expanding into a new venture called WeLive, up and running in New York and Washington DC, set to open for business in Seattle, and also planned for Tel Aviv. If accommodation is proving hard to find, you need company, and your life as a freelance means you have no permanent workplace where you can meet like-minded people, here is a solution: a range of tiny studio flats and slightly bigger dwellings, built around communal areas, kitchens and laundrettes – in the same building as WeWork office space.

Miguel McKelvey, one of the company’s two founders, has said that the idea is partly aimed at people who are “always working or always semi-working”. The mountain of press coverage this innovation has sparked includes a telling quote from one euphoric resident in Manhattan: “You just roll out of bed, go down the elevator and get to work.” This, apparently, is the future: despite its slow start, WeWork’s chief executive, Adam Neumann, insists that “WeLive is going to be a bigger business than WeWork”.

Four years after stories broke of employees who worked up to 90 hours a week living in camper vans at Google’s HQ in northern California, a WeLive-esque lifestyle will presumably take root at a new Google campus taking shape nearby, which will sit among 10,000 new “housing units”. Up the road, Facebook’s Willow Village development looks set to deliver something similar. Meanwhile, for tech high flyers lucky enough to have no fixed employer or workplace, as well as WeLive, there is an even more alluring option: Roam, which is pitched at “digital nomads”, and offers flexible “co-working and co-living” spaces in London, San Francisco, Miami, Tokyo and Ubud, in Indonesia. For upwards of $500 a week, such people can now wander around the world, mixing life and work – “two activities that quickly become indistinguishable within Roam’s confines”, as the New York Times put it.

It is telling that this blurring of work and leisure, and the fading-out of any meaningful notion of home, is reflected at every level of the tech industry – from shared houses that double as start-up “incubators” (see the hit HBO comedy Silicon Valley), through the co-working and co-living spaces springing up in urban China, to the factories in the same country where workers churning out iPhones sleep in dormitories. The erosion of any barrier between grafting and downtime is reflected in big tech’s innate insistence that we are “on” at all times – checking our feeds, sending emails, messaging colleagues. You see the same things even more clearly among rising numbers of networked homeworkers – translators, CV writers, IT contractors, data inputters – whose lives are often a very modern mixture of supposed flexibility, and day-to-day insecurity.

Marx and Engels said that the bourgeoisie could not exist “without constantly revolutionising the instruments of production, and thereby the relations of production, and with them the whole relations of society”; Tony Blair told us that the world of globalisation has “no custom or practice”, and gives rewards only to those “swift to adapt, slow to complain, open, willing and able to change”. And here, perhaps, is the ultimate proof. After a couple of centuries during which capitalism has recurrently tried to kill the inconvenient human need for domestic spaces where people can escape economic demands (witness such inventions as workers’ hostels and old-fashioned company towns), that same tendency is being newly dressed up as a matter of aspiration and personal freedom.

So what do we do? We urgently need a new Politics of Home. God knows, there will always be a market for expensive fads; and with prices for WeLive studios starting at over £2,300 a month, more fool anyone who takes the bait. But, as proved by other developments in the US and China – and a trailblazing scheme in London (at Old Oak, in Willesden) where cramped en suite rooms and access to a workspace can be had for a relatively affordable £245 a week – the worldwide push towards mixing up co-working and co-living highlights a big issue: the dire shortage of affordable urban housing.

More generally, the need for a distinction between work and downtime should enter the political vocabulary as a fundamental right, and the organisations dedicated to trying to enforce it – most notably, the network of small freelance unions that are dotted across Europe and the US – need to be encouraged and assisted.

It is time, too, that we begin to understand that the great wave of popular resentment sweeping across advanced societies is partly about the way the modern economy shreds some of people’s most basic emotional attachments. We all know the modern rules: millions of people have to leave where they grew up to find even halfway dependable work; and they find that creating any kind of substitute home somewhere new is impossible. For people at the bottom of the economic hierarchy, life proves to be unendingly precarious and often itinerant. For those slightly further up, the best available option seems to be a version of the student lifestyle that extends well into your 30s.

For all that some people seem to luxuriate in the weightlessness this fosters, it is surely no way to spend any sizable share of your adult life. And what if you want to push beyond footloose living and start a family?

It is a token of the surreal future some people want to push us towards that WeWork may have the beginnings of an answer to that question, albeit for the few people who can afford it. The company has recently spawned an educational offshoot called WeGrow (so far focused on a private elementary school in New York) that teaches kids a range of skills including mindfulness and “conscious entrepreneurship”. But the idea is apparently to put WeGrow schools in WeWork properties across the world, so digital nomads can carry their disorientated offspring from place to place, and ensure they have just as flimsy an idea of home as their parents do.

Not for the first time, you may well read this stuff and wonder: whose utopia is this?

• John Harris is a Guardian columnist

Image copyrightGetty Images

A new leak of documents from the offshore service provider at the centre of the Panama Papers scandal reveals the company could not identify the owners of up to three quarters of companies it administered.

Two months after becoming aware of the data breach, Mossack Fonseca was unable to identify the beneficial owners of more than 70% of 28,500 active companies in the British Virgin Islands (BVI) as well as 75% of companies in Panama, according to new documents seen by BBC Panorama.

At the time, BVI law permitted corporate service providers to rely upon intermediaries, banks, legal firms and other offshore service providers overseas to check the identities of the owners, although they were required to provide information if requested by the authorities.

The firm was fined $440,000 (£333,000) by the BVI Financial Services Commission in November 2016 for regulatory and legal infringements of anti-money laundering and other laws.

The new leak contains 1.2 million documents dating from before the Panama Papers went public in April 2016 to December 2017. The data was obtained by Suddeutsche Zeitung which shared it with the International Consortium of Investigative Journalists (ICIJ).

Email correspondence from clients and intermediaries reveal the reaction to the leak, Mossack Fonseca's desperate attempts to close gaps in their record keeping, and their difficulties doing so. Responding to questions from the firm, one Swiss wealth manager said: "THE CLIENT HAS DISAPPEARED! I CANT FIND HIM ANYMORE!!!!!!"

Other correspondence points to the primary reasons many clients were using offshore structures.

One Uruguayan financial planner commented: "…the main purpose of this type of structure has been broken: confidentiality".

Another intermediary wrote: "… the names of our customers have been known by the authorities of their countries. Thanks to Mossack, customers have to pay incomes taxes."

According to Margaret Hodge MP, former chair of the UK parliament's public accounts committee: "This is simply further proof, if any were needed, why we absolutely must have public registers of beneficial owners if we are to stamp out money laundering, tax avoidance, tax evasion and other crimes."

In the wake of the Panama Papers and under pressure from the UK government, the BVI and other British Overseas Territories have established registers of beneficial ownership for companies in their jurisdictions. But the register relies upon offshore service providers, like Mossack Fonseca, to provide the information. And it is only directly accessible to authorities in the BVI - the public cannot see it.

The BVI denies the register is secret and says it is accessible to the authorities and relevant UK authorities on request within one hour. They say they have been at the forefront of global transparency initiatives.

A spokesman said: "A public register is not a silver bullet. It is not about who can see the information, it's about the information being verified, accurate, and therefore useful to law enforcement. A verified register is a far more robust and effective approach to ensure transparency than an unverified public register."

The BVI has resisted pressure for a public register open to scrutiny.

Last month the UK government adopted an amendment to its Sanctions and Anti-Money Laundering Bill to force overseas territories to establish public registers by 2020. The BVI government is considering a legal challenge.

Margaret Hodge, who tabled the amendment with former Conservative minister Andrew Mitchell says: "I would be astounded if the BVI government chooses to waste their own taxpayers' money litigating against the British parliament - now supported by the British government."

A spokesman for BVI Finance said that legal advice was "that the imposition of a public register raises serious constitutional and human rights issues. As we have consistently stated, our position is that we will not introduce public registers until they become a global standard".

The Panama Papers investigation went public in April 2016 when the BBC and more than 100 other media organisations started publishing stories emanating from 11.5 million documents leaked from Mossack Fonseca.

The investigation was organised by the ICIJ and involved journalists from 76 countries scrutinising internal emails and corporate documents passed to the ICIJ by Suddeutsche Zeitung.

The German newspaper obtained the data from an anonymous source called "Jon Doe".

The UK investigation was led by BBC Panorama and the Guardian newspaper.

Revelations included documents linking Russian President Vladimir Putin's close friend, the cellist Sergi Roldugin, to suspected money laundering, Fifa corruption, sanctions busting and systemic tax dodging.

Then Prime Minister David Cameron faced a barrage of questioning over his late father's offshore investment fund and the prime minister of Iceland was forced to resign following demonstrations about his failure to declare an offshore interest.

Pakistani Prime Minister Nawaz Sharif resigned last year as a result of an investigation prompted by revelations about his family's offshore financial affairs.

Panamanian police raided Mossack Fonseca's offices and by the end of 2016 governments and companies in 79 countries had launched 150 inquiries, audits or investigations into the firm, companies it worked with, or its clients.

The new documents show how authorities from around the world, including the UK's Serious Fraud Office, contacted Mossack Fonseca demanding information about individuals and companies identified in the leak.

Mossack Fonseca's documents have since been obtained by a number of tax authorities including HM Revenue and Customs which paid an undisclosed sum for the leaked data.

In November 2017, the UK tax authority revealed there were 66 ongoing investigations related to the Panama Papers and that they expected to recover £100m in tax.

Three months ago Mossack Fonseca announced it was closing down citing reputational damage and the actions of the Panamanian authorities. In a statement this month, the company's founders said that neither they, the firm, nor its employees, were "involved in unlawful acts".

Last updated: December 21, 2017 | See all Documentation

Sometimes people want to get a certificate for the hostname “localhost”, either for use in local development, or for distribution with a native application that needs to communicate with a web application. Let’s Encrypt can’t provide certificates for “localhost” because nobody uniquely owns it, and it’s not rooted in a top level domain like “.com” or “.net”. It’s possible to set up your own domain name that happens to resolve to 127.0.0.1, and get a certificate for it using the DNS challenge. However, this is generally a bad idea and there are better options.

If you’re developing a web app, it’s useful to run a local web server like Apache or Nginx, and access it via http://localhost:8000/ in your web browser. However, web browsers behave in subtly different ways on HTTP vs HTTPS pages. The main difference: On an HTTPS page, any requests to load JavaScript from an HTTP URL will be blocked. So if you’re developing locally using HTTP, you might add a script tag that works fine on your development machine, but breaks when you deploy to your HTTPS production site. To catch this kind of problem, it’s useful to set up HTTPS on your local web server. However, you don’t want to see certificate warnings all the time. How do you get the green lock locally?

The best option: Generate your own certificate, either self-signed or signed by a local root, and trust it in your operating system’s trust store. Then use that certificate in your local web server. See below for details.

Sometimes developers want to offer a downloadable native app that can be used alongside a web site to offer extra features. For instance, the Dropbox and Spotify desktop apps scan for files from across your machine, which a web app would not be allowed to do. One common approach is for these native apps to offer a web service on localhost, and have the web app make requests to it via XMLHTTPRequest (XHR) or WebSockets. The web app almost always uses HTTPS, which means that browsers will forbid it from making XHR or WebSockets requests to non-secure URLs. This is called Mixed Content Blocking. To communicate with the web app, the native app needs to provide a secure web service.

Fortunately, modern browsers consider“http://127.0.0.1:8000/" to be a“potentially trustworthy” URL because it refers to a loopback address. Traffic sent to 127.0.0.1 is guaranteed not to leave your machine, and so is considered automatically secure against network interception. That means if your web app is HTTPS, and you offer a native app web service on 127.0.0.1, the two can happily communicate via XHR. Unfortunately, localhost doesn’t yet get the same treatment. Also, WebSockets don’t get this treatment for either name.

You might be tempted to work around these limitations by setting up a domain name in the global DNS that happens to resolve to 127.0.0.1 (for instance, localhost.example.com), getting a certificate for that domain name, shipping that certificate and corresponding private key with your native app, and telling your web app to communicate with https://localhost.example.com:8000/ instead of http://127.0.0.1:8000/.Don’t do this. It will put your users at risk, and your certificate may get revoked.

By introducing a domain name instead of an IP address, you make it possible for an attacker to Man in the Middle (MitM) the DNS lookup and inject a response that points to a different IP address. The attacker can then pretend to be the local app and send fake responses back to the web app, which may compromise your account on the web app side, depending on how it is designed.

The successful MitM in this situation is possible because in order to make it work, you had to ship the private key to your certificate with your native app. That means that anybody who downloads your native app gets a copy of the private key, including the attacker. This is considered a compromise of your private key, and your Certificate Authority (CA) is required to revoke your certificate if they become aware of it. Many native apps have had their certificates revoked for shipping their private key.

Unfortunately, this leaves native apps without a lot of good, secure options to communicate with their corresponding web site. And the situation may get trickier in the future if browsers further tighten access to localhost from the web.

Also note that exporting a web service that offers privileged native APIs is inherently risky, because web sites that you didn’t intend to authorize may access them. If you go down this route, make sure to read up on Cross-Origin Resource Sharing, use Access-Control-Allow-Origin, and make sure to use a memory-safe HTTP parser, because even origins you don’t allow access to can send preflight requests, which may be able to exploit bugs in your parser.

Anyone can make their own certificates without help from a CA. The only difference is that certificates you make yourself won’t be trusted by anyone else. For local development, that’s fine.

The simplest way to generate a private key and self-signed certificate for localhost is with this openssl command:

openssl req -x509 -out localhost.crt -keyout localhost.key \
  -newkey rsa:2048 -nodes -sha256 \
  -subj '/CN=localhost' -extensions EXT -config <( \
   printf "[dn]\nCN=localhost\n[req]\ndistinguished_name = dn\n[EXT]\nsubjectAltName=DNS:localhost\nkeyUsage=digitalSignature\nextendedKeyUsage=serverAuth")

You can then configure your local web server with localhost.crt and localhost.key, and install localhost.crt in your list of locally trusted roots.

If you want a little more realism in your development certificates, you can useminica to generate your own local root certificate, and issue end-entity (aka leaf) certificates signed by it. You would then import the root certificate rather than a self-signed end-entity certificate.

You can also choose to use a domain with dots in it, like “www.localhost”, by adding it to /etc/hosts as an alias to 127.0.0.1. This subtly changes how browsers handle cookie storage.

In fact, both Chrome and Firefox have tough security. Both include sandboxing, which isolates processes of the browser so a harmful website doesn’t infect other parts of your machine. So if you loaded a website with malicious code, it would be contained within the webpage so it couldn’t infect your files, webcam and microphone.

Google said there was one thing it could do better on: the inclusion of privacy settings to block tracking technology, similar to the tools that Firefox includes.

“I think that’s something that we can improve on,” Ms. Tabriz said. “Firefox has some settings that we’re also exploring.”

Speed and Battery Tests

Which browser is faster?

Some benchmark websites, which determine the speed of a browser by measuring the responsiveness of different web elements, say Chrome is faster. But some other benchmark sites say Firefox is faster. In my anecdotal testing as someone who juggles more than a dozen web tabs at a time, both were very speedy. Let’s call it a draw.

Mozilla’s promise that Firefox consumes less computer memory raises hopes that it should also use less battery life. Yet in my tests on a laptop running a script that automatically reloaded the top 10 news sites, Firefox lasted only a few minutes longer than Chrome before the battery was depleted. On another test, which involved streaming a Netflix video on a loop on each browser, the battery lasted about 20 minutes longer when the Chrome browser was used.

Resurrection Is Just Beginning

Firefox is the No. 2 computer browser, with about 12 percent of the desktop browser market, lagging far behind Chrome, which has about 67 percent, according to StatCounter. Microsoft’s Internet Explorer and Apple’s Safari browsers are even farther behind in the desktop market, with Explorer’s share about 7 percent and Safari’s about 5.5 percent. On Android phones, the Chrome browser is still far more popular than Firefox’s mobile browser. And only lightweight versions of Firefox are available for Apple’s iOS devices.

Yet the path forward for Mozilla looks increasingly promising for consumers.

In addition to the normal Firefox browser, Mozilla offers Firefox Focus, a privacy-centric mobile browser that blocks trackers by default and purges your web browsing history as soon as you close out of a page.

A trainee day trader in France is suing a British brokerage for an amount comparable to almost its entire annual revenue after it seized the €10m profit he made using what he initially thought was a demonstration version of its platform.

Harouna Traoré opened a €20,000 account at Valbury Capital, a UK-based brokerage, last summer after using a dummy version of its platform to learn how to trade equity futures as a retail investor on a trading course in Paris.

A couple of weeks later, he was practising trading at home on what he believed to be the demo version — placing €1bn of orders for European and US equity futures — before realising that it was the live platform and he had run up a loss of more than €1m.

He continued trading, eventually building up a $5bn position in US equity futures and turning the loss into a profit of more than €10m. “I could only think of my family,” said Mr Traoré, who is married with two children. “I was stressed.” After he called Valbury a few days later to explain what had happened, the brokerage told him he had breached his contract and his positions were “void and cancelled”.

In January, he filed a writ of summons in the Pontoise district court, north of Paris, claiming breach of contract and negligence by the British brokerage and calling for it to pay him the €10m he says it owes him.

Valbury, which is owned by the eponymous Indonesian financial services group, denies any wrongdoing and is preparing to file its initial submission next week. It is expected to argue that Mr Traoré is not a consumer, but a financial services professional, so the case should not be heard in France, where he would benefit from greater consumer protection.

Robert Falkner, partner at Reed Smith, the law firm representing Valbury, said: “We are familiar with the spurious allegations made by the French arcade trader Mr Traoré (a seasoned market risk analyst formerly employed by Reuters) which are strongly denied as wholly without merit and will be vigorously contested.”

“This matter is now before the courts so that we consider it inappropriate to comment further,” said Mr Falkner, adding that Valbury had kept its regulators at the UK Financial Conduct Authority “fully informed”.

Valbury is expected to point out that Mr Traoré said in his application to open an account that he had traded futures and options frequently. Mr Traoré admitted that he had “tried to embellish my trading experience and professional qualifications at the time, as I thought my application might otherwise not go through as easily”.

According to the court filing by Mr Traoré, which has been seen by the Financial Times, Valbury told him that it had treated the trades he carried out as a “manifest error” because he had thought he was using its demo platform and had not intended to place real orders. The brokerage also told him that he had breached his trading limits.

Mr Traoré’s lawyers at Linklaters said in the filing that the 41-year-old had no prior experience of financial markets and previously worked at Thomson Reuters, selling performance analysis software to investors, before being made redundant last year.

Therefore, they said, he should be considered a consumer and the case should be heard in France. His lawyers also said Mr Traoré should have been prevented from trading such large amounts by preset trading limits that could have been imposed by Valbury.

They disputed Valbury’s suggestion that his orders were a “manifest error” — the definition usually given to fat-finger trading mistakes — because most of the profits were only made once he realised he was trading on the live platform.

The stakes are high for Valbury, which made £9.88m of revenue in the year to December 2016 down from £11.7m the previous year. It reported its third consecutive annual loss of £455,405 in 2016 — its last set of publicly filed accounts.

Mark Hanney, chief executive of Valbury, has previously worked at several other trading firms and spent five years as financial director of Refco Trading Services Ltd, the UK arm of the collapsed US brokerage. Mr Hanney declined to comment.

The FCA declined to comment.

In 2015, a worried father asked Rhema Vaithianathan a question that still weighs on her mind. A small crowd had gathered in a basement room in Pittsburgh, Pennsylvania, to hear her explain how software might tackle child abuse. Each day, the area’s hotline receives dozens of calls from people who suspect that a child is in danger; some of these are then flagged by call-centre staff for investigation. But the system does not catch all cases of abuse. Vaithianathan and her colleagues had just won a half-million-dollar contract to build an algorithm to help.

Vaithianathan, a health economist who co-directs the Centre for Social Data Analytics at the Auckland University of Technology in New Zealand, told the crowd how the algorithm might work. For example, a tool trained on reams of data — including family backgrounds and criminal records — could generate risk scores when calls come in. That could help call screeners to flag which families to investigate.

After Vaithianathan invited questions from her audience, the father stood up to speak. He had struggled with drug addiction, he said, and social workers had removed a child from his home in the past. But he had been clean for some time. With a computer assessing his records, would the effort he’d made to turn his life around count for nothing? In other words: would algorithms judge him unfairly?

Vaithianathan assured him that a human would always be in the loop, so his efforts would not be overlooked. But now that the automated tool has been deployed, she still thinks about his question. Computer calculations are increasingly being used to steer potentially life-changing decisions, including which people to detain after they have been charged with a crime; which families to investigate for potential child abuse, and — in a trend called ‘predictive policing’ — which neighbourhoods police should focus on. These tools promise to make decisions more consistent, accurate and rigorous. But oversight is limited: no one knows how many are in use. And their potential for unfairness is raising alarm. In 2016, for instance, US journalists argued that a system used to assess the risk of future criminal activity discriminates against black defendants.

“What concerns me most is the idea that we’re coming up with systems that are supposed to ameliorate problems [but] that might end up exacerbating them,” says Kate Crawford, co-founder of the AI Now Institute, a research centre at New York University that studies the social implications of artificial intelligence.

With Crawford and others waving red flags, governments are trying to make software more accountable. Last December, the New York City Council passed a bill to set up a task force that will recommend how to publicly share information about algorithms and investigate them for bias. This year, France’s president, Emmanuel Macron, has said that the country will make all algorithms used by its government open. And in guidance issued this month, the UK government called for those working with data in the public sector to be transparent and accountable. Europe’s General Data Protection Regulation (GDPR), which came into force at the end of May, is also expected to promote algorithmic accountability.

In the midst of such activity, scientists are confronting complex questions about what it means to make an algorithm fair. Researchers such as Vaithianathan, who work with public agencies to try to build responsible and effective software, must grapple with how automated tools might introduce bias or entrench existing inequity — especially if they are being inserted into an already discriminatory social system.

The questions that automated decision-making tools raise are not entirely new, notes Suresh Venkatasubramanian, a theoretical computer scientist at the University of Utah in Salt Lake City. Actuarial tools for assessing criminality or credit risk have been around for decades. But as large data sets and more-complex models become widespread, it is becoming harder to ignore their ethical implications, he says. “Computer scientists have no choice but to be engaged now. We can no longer just throw the algorithms over the fence and see what happens.”

Fairness trade-offs

When officials at the Department of Human Services in Allegheny County, where Pittsburgh is located, called in 2014 for proposals for an automated tool, they hadn’t yet decided how to use it. But they knew they wanted to be open about the new system. “I’m very against using government money for black-box solutions where I can’t tell my community what we’re doing,” says Erin Dalton, deputy director of the department’s Office of Data Analysis, Research and Evaluation. The department has a centralized data warehouse, built in 1999, that contains a wealth of information about individuals — including on housing, mental health and criminal records. Vaithianathan’s team put in an impressive bid to focus on child welfare, Dalton says.

The Allegheny Family Screening Tool (AFST) launched in August 2016. For each phone call to the hotline, call-centre employees see a score between 1 and 20 that is generated by the automated risk-assessment system, with 20 corresponding to a case designated as highest risk. These are families for which the AFST predicts that children are most likely to be removed from their homes within two years, or to be referred to the county again because a caller has suspected abuse (the county is in the process of dropping this second metric, which does not seem to closely reflect the cases that require further investigation).

An independent researcher, Jeremy Goldhaber-Fiebert at Stanford University in California, is still assessing the tool. But Dalton says preliminary results suggest that it is helping. The cases that call-centre staff refer to investigators seem to include more instances of legitimate concern, she says. Call screeners also seem to be making more consistent decisions about cases that have similar profiles. Still, their decisions don’t necessarily agree with the algorithm’s risk scores; the county is hoping to bring the two into closer alignment.

As the AFST was being deployed, Dalton wanted more help working out whether it might be biased. In 2016, she enlisted Alexandra Chouldechova, a statistician at Carnegie Mellon University in Pittsburgh, to analyse whether the software was discriminating against particular groups. Chouldechova had already been thinking about bias in algorithms — and was about to weigh in on a case that has triggered substantial debate over the issue.

In May that year, journalists at the news website ProPublica reported on commercial software used by judges in Broward County, Florida, that helps to decide whether a person charged with a crime should be released from jail before their trial. The journalists said that the software was biased against black defendants. The tool, called COMPAS, generated scores designed to gauge the chance of a person committing another crime within two years if released.

The ProPublica team investigated COMPAS scores for thousands of defendants, which it had obtained through public-records requests. Comparing black and white defendants, the journalists found that a disproportionate number of black defendants were ‘false positives’: they were classified by COMPAS as high risk but subsequently not charged with another crime.

The developer of the algorithm, a Michigan-based company called Northpointe (now Equivant, of Canton, Ohio), argued that the tool was not biased. It said that COMPAS was equally good at predicting whether a white or black defendant classified as high risk would reoffend (an example of a concept called ‘predictive parity’). Chouldechova soon showed that there was tension between Northpointe’s and ProPublica’s measures of fairness¹. Predictive parity, equal false-positive error rates, and equal false-negative error rates are all ways of being ‘fair’, but are statistically impossible to reconcile if there are differences across two groups — such as the rates at which white and black people are being rearrested (see ‘How to define ‘fair’’). “You can’t have it all. If you want to be fair in one way, you might necessarily be unfair in another definition that also sounds reasonable,” says Michael Veale, a researcher in responsible machine learning at University College London.

In fact, there are even more ways of defining fairness, mathematically speaking: at a conference this February, computer scientist Arvind Narayanan gave a talk entitled ‘21 fairness definitions and their politics’ — and he noted that there were still others. Some researchers who have examined the ProPublica case, including Chouldechova, note that it’s not clear that unequal error rates are indicative of bias. They instead reflect the fact that one group is more difficult to make predictions about than another, says Sharad Goel, a computer scientist at Stanford. “It turns out that that’s more or less a statistical artefact.”

For some, the ProPublica case highlights the fact that many agencies lack resources to ask for and properly assess algorithmic tools. “If anything, what it’s showing us is that the government agency who hired Northpointe did not give them a well-defined definition to work with,” says Rayid Ghani, who directs the Center for Data Science and Public Policy at the University of Chicago, Illinois. “I think that governments need to learn and get trained in how to ask for these systems, how to define the metrics they should be measuring and to make sure that the systems they are being given by vendors, consultants and researchers are actually fair.”

Allegheny County’s experience shows how difficult it is to navigate these questions. When Chouldechova, as requested, began digging through the Allegheny data in early 2017, she found that its tool also suffered similar statistical imbalances. The model had some “pretty undesirable properties”, she says. The difference in error rates was much higher than expected across race and ethnicity groups. And, for reasons that are still not clear, white children that the algorithm scored as at highest risk of maltreatment were less likely to be removed from their homes than were black children given the highest risk scores². Allegheny and Vaithianathan’s team are currently considering switching to a different model. That could help to reduce inequities, says Chouldechova.

Although statistical imbalances are a problem, a deeper dimension of unfairness lurks within algorithms — that they might reinforce societal injustices. For example, an algorithm such as COMPAS might purport to predict the chance of future criminal activity, but it can only rely on measurable proxies, such as being arrested. And variations in policing practices could mean that some communities are disproportionately targeted, with people being arrested for crimes that might be ignored in other communities. “Even if we are accurately predicting something, the thing we are accurately predicting might be the imposition of injustice,” says David Robinson, a managing director at Upturn, a non-profit social-justice organization in Washington DC. Much would depend on the extent to which judges rely on such algorithms to make their decisions — about which little is known.

Allegheny’s tool has come under criticism along similar lines. Writer and political scientist Virginia Eubanks has argued that, irrespective of whether the algorithm is accurate, it is acting on biased inputs, because black and biracial families are more likely to be reported to hotlines. Furthermore, because the model relies on public-services information in the Allegheny system — and because the families who used such services are generally poor — the algorithm unfairly penalizes poorer families by subjecting them to more scrutiny. Dalton acknowledges that the available data are a limitation, but she thinks the tool is needed. “The unfortunate societal issue of poverty does not negate our responsibility to improve our decision-making capacity for those children coming to our attention,” the county said in a response to Eubanks, posted on the AFST website earlier this year.

Transparency and its limits

Although some agencies build their own tools or use commercial software, academics are finding themselves in demand for work on public-sector algorithms. At the University of Chicago, Ghani has been working with a range of agencies, including the public-health department of Chicago on a tool to predict which homes might harbour hazardous lead. In the United Kingdom, researchers at the University of Cambridge have worked with police in County Durham on a model that helps to identify who to refer to intervention programmes, as an alternative to prosecution. And Goel and his colleagues this year launched the Stanford Computational Policy Lab, which is conducting collaborations with government agencies, including the San Francisco District Attorney’s office. Partnerships with outside researchers are crucial, says Maria McKee, an analyst at the district attorney’s office. “We all have a sense of what is right and what is fair,” she says. “But we often don’t have the tools or the research to tell us exactly, mechanically, how to get there.”

There is a large appetite for more transparency, along the lines adopted by Allegheny, which has engaged with stakeholders and opened its doors to journalists. Algorithms generally exacerbate problems when they are “closed loops that are not open for algorithmic auditing, for review, or for public debate”, says Crawford at the AI Now Institute. But it is not clear how best to make algorithms more open. Simply releasing all the parameters of a model won’t provide much insight into how it works, says Ghani. Transparency can also conflict with efforts to protect privacy. And in some cases, disclosing too much information about how an algorithm works might allow people to game the system.

One big obstacle to accountability is that agencies often do not collect data on how the tools are used or their performance, says Goel. “A lot of times there’s no transparency because there’s nothing to share.” The California legislature, for instance, has a draft bill that calls for risk-assessment tools to help reduce how often defendants must pay bail — a practice that has been criticized for penalizing lower-income defendants. Goel wants the bill to mandate that data are collected on instances when judges disagree with the tool and on specific details, including outcomes, of every case. “The goal is fundamentally to decrease incarceration while maintaining public safety,” he says, “so we have to know — is that working?”

Crawford says that a range of ‘due process’ infrastructure will be needed to ensure that algorithms are made accountable. In April, the AI Now Institute outlined a framework³ for public agencies interested in responsible adoption of algorithmic decision-making tools; among other things, it called for soliciting community input and giving people the ability to appeal decisions made about them.

Many are hoping that laws could enforce such goals. There is some precedent, says Solon Barocas, a researcher who studies ethics and policy issues around artificial intelligence at Cornell University in Ithaca, New York. In the United States, some consumer-protection rules grant citizens an explanation when an unfavourable decision is made about their credit⁴. And in France, legislation that gives a right to explanation and the ability to dispute automated decisions can be found as early as the 1970s, says Veale.

The big test will be Europe’s GDPR, which entered into force on 25 May. Some provisions — such as a right to meaningful information about the logic involved in cases of automated decision-making — seem to promote algorithmic accountability. But Brent Mittelstadt, a data ethicist at the Oxford Internet Institute, UK, says the GDPR might actually hamper it by creating a “legal minefield” for those who want to assess fairness. The best way to test whether an algorithm is biased along certain lines — for example, whether it favours one ethnicity over another — requires knowing the relevant attributes about the people who go into the system. But the GDPR’s restrictions on the use of such sensitive data are so severe and the penalties so high, Mittelstadt says, that companies in a position to evaluate algorithms might have little incentive to handle the information. “It seems like that will be a limitation on our ability to assess fairness,” he says.

The scope of GDPR provisions that might give the public insight into algorithms and the ability to appeal is also in question. As written, some GDPR rules apply only to systems that are fully automated, which could exclude situations in which an algorithm affects a decision but a human is supposed to make the final call. The details, Mittelstadt says, should eventually be clarified in the courts.

Auditing algorithms

Meanwhile, researchers are pushing ahead on strategies for detecting bias in algorithms that haven’t been opened up for public scrutiny. Firms might be unwilling to discuss how they are working to address fairness, says Barocas, because it would mean admitting that there was a problem in the first place. Even if they do, their actions might ameliorate bias but not eliminate it, he says. “So any public statement about this will also inevitably be an acknowledgment that the problem persists.” But in recent months, Microsoft and Facebook have both announced the development of tools to detect bias.

Some researchers, such as Christo Wilson, a computer scientist at Northeastern University in Boston, try to uncover bias in commercial algorithms from the outside. Wilson has created mock passengers who purport to be in search of Uber taxi rides, for example, and has uploaded dummy CVs to a jobs website to test for gender bias. Others are building software that they hope could be of general use in self-assessments. In May, Ghani and his colleagues released open-source software called Aequitas to help engineers, policymakers and analysts to audit machine-learning models for bias. And mathematician Cathy O’Neil, who has been vocal about the dangers of algorithmic decision-making, has launched a firm that is working privately with companies to audit their algorithms.

Some researchers are already calling for a step back, in criminal-justice applications and other areas, from a narrow focus on building algorithms that make forecasts. A tool might be good at predicting who will fail to appear in court, for example. But it might be better to ask why people don’t appear and, perhaps, to devise interventions, such as text reminders or transportation assistance, that might improve appearance rates. “What these tools often do is help us tinker around the edges, but what we need is wholesale change,” says Vincent Southerland, a civil-rights lawyer and racial-justice advocate at New York University’s law school. That said, the robust debate around algorithms, he says, “forces us all to ask and answer these really tough fundamental questions about the systems that we’re working with and the ways in which they operate”.

Vaithianathan, who is now in the process of extending her child-abuse prediction model to Douglas and Larimer counties in Colorado, sees value in building better algorithms, even if the overarching system they are embedded in is flawed. That said, “algorithms can’t be helicopter-dropped into these complex systems”, she says: they must be implemented with the help of people who understand the wider context. But even the best efforts will face challenges, so in the absence of straight answers and perfect solutions, she says, transparency is the best policy. “I always say: if you can’t be right, be honest.”

Intel Corp. removed Brian Krzanich as chief executive officer after the chipmaker learned he had previously had a consensual relationship with an employee, a violation of the company’s policies.

Chief Financial Officer Robert Swan was made interim CEO while the board searches for a permanent replacement, the company said in a statement Thursday.

“The board believes strongly in Intel’s strategy and we are confident in Bob Swan’s ability to lead the company as we conduct a robust search for our next CEO,” the company said.

Krzanich, 58, who moved up the ranks at Intel over more than three decades, has been CEO since 2013 and oversaw the company through an era of intense competition and consolidation in the chip industry. He had been trying to remake Intel into a more general provider of chips, expanding into new markets such as industrial systems and self-driving cars, with the 2017 purchase of Mobileye for $15.3 billion. The mission remains a work in progress; the data-center chip business is still the biggest sales engine and will determine Intel’s success for the foreseeable future.

Known as “BK” at Intel because of the difficulties some have in pronouncing his surname (Kris-an-itch), he worked his way up Intel’s ranks as a factory manager. He joined Intel in 1982 in New Mexico as an engineer, just as the personal-computer industry was beginning to take off.

Swan, the former chief financial officer of Ebay Inc., has been Intel’s CFO since October 2016. He oversees finance, information technology and corporate strategy. He previously was CEO of Webvan Group Inc.

Intel also raised its second-quarter revenue and profit forecast Thursday.

Intel climbed 1.7 percent $54.38 at 9:20 a.m. in New York. The company rallied 16 percent this year through Wednesday, beating the Dow Jones Industrial Average, which is little changed.

(Updates with company quote in third paragraph.)

The Altai mountain region of Central Asia is a rugged and remote place. Right in the center of the continental landmass, it forms a crossroads between the Kazakh steppes, the snow forests of Siberia and the arid plains of Mongolia. It’s a landscape of granite, forced up by the inch-a-year collision of the Indian tectonic plate with Asia, then carved out over millions of years by streams of snowmelt. Siberian Ibex wander here along with musk deer feeding on the lichenous rocks and brown bears that follow the retreating snow fields in spring.

This might be one of the most remote places on earth, little accessible by road, but its peace is routinely broken in the most dramatic way. That’s because the Altai region sits right beneath the main flight path of the oldest, largest and busiest spaceport in the world: the Baikonur Cosmodrome. Debris from each rocket launch rains down on these remote hills, and the people of the region are forced to make a living among the falling scraps.

Built in 1955 in the grasslands of southern Kazakhstan, the Baikonur Cosmodrome has been the launch site for many historic missions. Earth’s first artificial satellite Sputnik 1 launched here, and Yuri Gagarin’s 1961 flight as the first human into space also took off from Baikonur. Today it’s home to space missions from around the world, including monthly commercial, scientific and military launches. Since the retirement of the US shuttle program, Russian Soyuz capsules launched from Baikonur have also become the only remaining lifeline to resupply and re-man the International Space Station (ISS). Between 2006 and 2018, NASA paid the Russian space agency Roscosmos approximately $3.4 billion USD to ferry astronauts to the station.

Even farm tools and sledges for children have been built out of the fragments of rocket hulls.

All this traffic creates a huge amount of debris along the corridor of rocket flightpaths. To lift a 6.3 ton satellite into geostationary orbit requires a 4-stage Russian Proton rocket weighing nearly 700 tons. As the rocket streaks off in its northeasterly flight path, booster rockets peel away from the craft in 3 stages and fall back down to Earth. While rockets launched from NASA’s Kennedy Space Center in Florida are able to drop their boosters relatively harmlessly into the Atlantic ocean, Baikonur is about as as far from the ocean as it is possible to get. That means the discarded stages of Russian rockets tumble back down on dry land. The first stage usually falls within 90 kilometers of the launchpad, but the second stage flies for a full 14 minutes more, and lands with potentially devastating impact 1,000 kilometers away in the Altai.

Russian media estimates that over 2,500 tons of space debris have rained down on the region since launches began in the 1950s, some segments as large as 10 meters in length. During the Soviet era, the USSR took great pains to recover Baikonur booster rockets, partially due to concerns about leaking secrets related to the space program. However, since the collapse of the Soviet Union, these pieces of space debris have increasingly been left to rust in the grasslands of Kazakhstan and the Altai mountains.

The launches have become a familiar sight for those who live beneath the flight paths. Pieces of falling debris look “like an angry red eye in the night,” one resident says. Then there is a great thundering sound, and “a small earthquake” shakes the ground. Roscosmos designates a narrow strip of land across the region in which the rocket stages are supposed to fall. Residents within this zone are given 24 hours’ notice of a launch to get themselves to safety, and it’s only outside this zone that people can claim compensation for damages. However, incidents outside the zone are far from uncommon. In 2008, a piece of debris measuring 4.5 meters in length even landed in a village, narrowly missing a house. Debris rains down even when launches go to plan; failures can have much more serious consequences. In 2011 an unmanned Progress 44 capsule atop a Soyuz-U rocket was headed to the ISS when it failed in the first 5 minutes of its launch. The rocket tumbled back down to Earth with its later stages still full of fuel, and hit the mountains of the Altai. The resulting explosion shattered windows up to 100 kilometers away, though no one was hurt.

While many residents fear the monthly rocket launches that pepper the region with debris, others see in it a unique opportunity. Resourceful scrap dealers wait for the announcement of rocket launches and then watch the sky with binoculars. They track the paths of debris and ride out on jeeps and even on horses to the crash sites. With little protective gear other than welding masks, they use blowtorches to strip the wrecks of their valuable light metals, alloys of titanium and aluminium, as well as other useful components like copper wire. Visitors to the region have reported seeing the roofs of chicken coops and sheds built with rocket parts still showing the original Proton insignia. Even farm tools and sledges for children have been built out of the fragments of rocket hulls.

It’s dangerous work for the scrap dealers. Rocket parts are frequently still burning when they arrive, leaching noxious vapours and setting off wildfires in the dry steppes. Nevertheless, the precarious economic reality of the region means that the extra source of income is indispensable. Here, a whole industry has sprung up around the waste of the world’s richest nations as it rains periodically from the sky.

Residents’ tendency to bring parts of the rockets back to their villages poses serious dangers, though. Rocket fuel, especially of the kind used by Russian and Chinese rockets, contains highly toxic components, and discarded rocket sections can have as much as 10 percent of their fuel remaining when they tumble back down to Earth. Of these, the most feared is heptyl, which is extremely toxic, and has been linked to cancers and birth defects. In 2008, a farmer in the Altai region filed for compensation when four of his horses died after toxic substances from rocket debris allegedly leached into the soil of their grazing lands.

The people of the Altai depend heavily on the land, growing crops, raising livestock and foraging for pine nuts in the virgin snow forests of Siberian cedar. Doctors in the region blame increasing health problems on the proliferation of space junk falling down to earth and the toxic compounds that make their way into the food and water. In 2005, a Nature study showed children in affected areas were twice as likely to develop endocrine and blood disorders, with rates of all other diseases markedly higher too. In some villages, virtually every child is reportedly born with jaundice. One farmer has even recounted seeing Siberian deer wandering the snow forests, blinded by toxic chemicals. The Russian government denies any evidence of a connection between the rocket debris and ongoing health problems, but they have elected to keep their research out of the public eye.

Despite its long history, Baikonur’s heyday might soon come to an end. Currently under construction is the Vostochny Cosmodrome, a new facility in the far east of Russia intended to reduce dependency on the landlocked Baikonur. At a cost of $7.5 billion USD, it might help to reduce the environmental impact of rocket launches on the people of the Altai. However, reports of corruption and the exploitation of workers has dogged the expensive project, and it will not begin to launch heavier rockets until 2021.

Today, the pockmarked and poisoned grasslands of the Altai seem like a striking metaphor for the world’s inequality and the costs of technological progress. While the richest nations send their citizens and machines to explore the further reaches of the solar system, their debris rains down on some of the world’s poorest. The region asks us to contemplate what moral price our progress comes at, and who in the end gets to decide this cost.

This article was originally published on DiscoverMagazine.com on June 7, 2018.

1. AnyChart News
2. Data Visualization with Elasticsearch and AnyChart JavaScript Charts — Integration Sample

Data Visualization with Elasticsearch and AnyChart JavaScript Charts — Integration Sample June 21st, 2018 by Irina Maximova

It’s always been important to AnyChart to make JavaScript charts run in HTML5 projects fast regardless of the users’ technology stack. Our AnyChart, AnyMap, AnyStock, and AnyGantt libraries for data visualization are compatible with all major technologies and platforms, and there is a bunch of different integration samples already available. All of them are a great help in making the process of chart creation more pleasant and less time-consuming. In this article, we will tell you how to integrate AnyChart JS Charts with Elasticsearch. To begin with, let’s find out what Elasticsearch is and what makes it special.

What is Elasticsearch

Elasticsearch is a full-text search and analytics engine based on Lucene. It has lots of advantages which make many companies rely on it to solve their problems. Elasticsearch is a distributed, RESTful search and analytics engine, it is a central storage of your data within Elastic Stack.

The engine is really fast and easy to set up. Its multi-tenancy feature allows you have the same machine running different indices. Moreover, Elasticsearch is trouble-free, it detects failures to keep your cluster (and your data) safe and available. It should also be noted here that all data types are welcome in Elasticsearch and it’s possible to combine many types of searches — structured, unstructured, geo, and metric ones.

Quick start

Let’s see how to make the sample work and enjoy the beauty of this integration. The first thing you need here is Elasticsearch installed. If you don’t have it, please, visit the Elasticsearch download page or install it with WGET utility using this install guide. You should have JDK 6 or above, and NPM – a package manager for Node.js.

To run Elasticsearch server and check its status, execute the following command in Elasticsearch installed folder on Unix systems:

$ bin/elasticsearch

Or this one on Windows:

$ bin\elasticsearch.bat

Elasticsearch is running now. You can access it at http://localhost:9200 in your web browser or get info with CURL:

$ curl http://localhost:9200/

To start this example, run the commands listed below.

First of all, clone the repository from github.com:

$ git clone git@github.com:anychart-integrations/elasticsearch-basic-sample.git

Then navigate to the repository folder:

$ cd elasticsearch-template

Install dependencies:

$ npm install

After that, put some test data to the Elasticsearch instance where ‘testbase’ is intended for an index, ‘dataset’ is for a type and ‘1’ is for an id:

$ curl -XPOST -H'Content-Type: application/json' http://localhost:9200/testbase/dataset/1 -d '{"data":[{"x":"Apples", "value": "128"},{"x":"Oranges", "value": "99"},{"x":"Lemons", "value": "54"},{"x":"Bananas", "value": "150"}] }'

And finally, run the sample:

$ npm start

Now when you open the browser at http://localhost:8080, you’ll see a Pie Chart sample that looks like this:

While Elasticsearch is running you may use commands to:

1) get the current mapping of index ‘testbase’:

$ curl -XGET -H'Content-Type: application/json' "http://localhost:9200/testbase/_mapping?pretty"

2) get the document from index ‘testbase’ with the type ‘dataset’ and id ‘1’:

$ curl -XGET "http://localhost:9200/testbase/dataset/1?pretty"

3) delete existing index ‘testbase’ from the Elasticsearch instance:

$ curl -XDELETE http://localhost:9200/testbase

The full source code of this Elasticsearch integration can be found on GitHub.

Sample code

To understand how we connect AnyChart library with the search and analytics engine Elasticsearch, you can look at the two most important files in our repository. These are index.html and server.js.

The index.html file is where we draw a chart there, get its data, set the title, and so on. Server.js starts the Express server that takes the data from Elasticsearch. When a user requests the Express server, it forms a static HTML page with the chart that contains the data from Elasticsearch and sends it to the user. The example of how the API allows Elasticsearch to work under Node.js can be found at The official low-level Elasticsearch client for Node.js and the browser.

We hope you found this article useful. Please check out other AnyChart integration templates. All the AnyChart integrations are released on AnyChart Integrations page on GitHub under the Apache 2.0 license and can be forked and changed in any way.

The collection of our integration samples tends to grow. And we will be happy if you send us your suggestions on what else you want to see in there. Stay tuned.

Issue Management is a process of identifying, reporting and resolving all such issues and bugs that may otherwise impair the progress of a project.

The process of managing issues, however, is not always a smooth ride. You have to ensure each issue is adequately reported on time, prioritized, logged into the database and appointed to a suitable person who has the optimum set of skills to fix it. Which is why Project Managers employ the help of an issue management software.

Here is our list of best issue management software tools that are on the rise in 2018:

Created way back in 1998 by the Mozilla Foundation, Bugzilla is practically the Godzilla of Issue tracking software. It is one of the most powerful platforms that help developers and teams alike, in bug tracking as well as project management. Bugzilla is a web-based tool whose expertise lies mainly in issue tracking and reporting. Even after decades, Bugzilla has not lost its charm, owing to its unparalleled features like advanced search capabilities and report charts that are automatically generated for scheduled reporting.

Key Features:

• Provides a display of complete bug/issue change history
• Enables users to attach and manage multiple issue related files on a given task
• Allows issues and bugs to be grouped into filters using various parameters.
• Provides a graphical representation of reports for a comprehensive analysis
• Contains email notifications that can be configured whenever there is a change in an issue
• Enables efficient time tracking of Issues and bugs
• Allows users to Save & Share Searches

Drawbacks:

• Limited ability to customize workflow and dashboard
• Limited visibility due to lack of scrum and Kanban boards
• Not a cloud-based server. Only a web-based server

Pricing:

• Open Source Tool. Completely Free!

Built for Agile teams and Agile management, Jira is a leading issue management software, that has acquired more than 75000 users across the globe. Jira’s extensive set of features allows teams of all sizes and nature to report, manage and track issues anywhere in a project’s lifecycle. It is because of this extensibility, that Jira renders itself ideal not just for the software development industry, but teams from all sorts of industries working to manage issues in their line of work.

Key Features

• Agile Issue tracking and reporting, to help add iterative value to the process of issue management
• Agile time tracking, that allows real-time insight to a team’s work and their performance on a sprint-to-sprint basis
• Maximum visibility with the help of Kanban boards and scrum boards that can be customized as per user convenience.
• Allows efficient transitioning and prioritization of issues via drag and drop feature across Kanban boards
• Robust APIs that facilitate numerous integrations with neighboring platforms and browsers.
• Provides unlimited bug fields with powerful advanced search and filters

Drawbacks

• Complex user-interface that complicates usability and often results in a long learning curve for teams
• The pricing plan is suitable for medium to large enterprises as opposed to small organizations
• Requires extensive configuration that makes it hard for users to make changes to workflows

Pricing

• Up to 10 Users – $10 monthly
• From 11-100 users – $7 per user/month

nTask is an online project management tool designed to streamline team collaboration within a single platform. It was created to simplify issue management with easy-to-maneuver features such as issue updates, issue assignees and change in issue status and priority.

In addition to Issue Management, nTask is an all-in-one tool that facilitates all necessary aspects of project management that include but are not limited to, meeting management, team management, and time management.

Key Features

• Easy to understand dashboard with issues assembled in a List view.
• A collection of filters to group and manage issues based on user preference
• Ability to link captured issues to relevant project or task.
• Allows users to define a status, priority and severity level for each issue
• Enables issue customization, so users can appoint and change the title of the issue, add details and attachments of any size and type.
• Assign multiple issues to multiple members for adequate issue tracking
• In-app notifications that arrive individually on each issue-pop up for immediate recognition

Drawbacks

• No integrations with neighboring tools and browsers.
• No Kanban or scrum boards for customizable visibility
• Issues can’t be deleted. They can only be switched to a ‘Closed’ status.

Pricing

• Free Plan – up to 10 members. 200 MB Storage
• $2.99 per user/month – unlimited access to all features.

Backlog is everything you need for a development workflow. In addition to effortless issue tracking, it enables teams to track and manage tasks as well as projects. Tasks can be divided into sub-tasks, that can then be assigned to one or more team members. With Backlog, teams can flawlessly manage end-to-end development of their product by assigning suitable members to capture issues and work on resolving them as early as possible.

Key Features

• Easy and systemic documentation of issues with parameters such as title, description, and attachments that help define a bug for quick assessment
• Facilitates Issue prioritization and issue assigning. Based on the level of urgency, teams can streamline their workload, change due dates for issues and stay in sync with which issues needs immediate attention
• Targeted notifications and issue-specific commenting allow the right people to stay informed on instant alerts and changes made to the issues created.
• Git branch view, that facilitates issue updates, fix-releases, and pulls requests all on a single chart.
• Multiple integrations with a variety of other tools
• Drag and Drop feature for smooth accessibility

Drawbacks

• Majority of advanced features are only accessible in Paid version

Pricing

• Free plan – up to 10 users. 100 MB online storage
• $20/month – up to 30 users. 1 GB storage

ReQtest is another cloud-based project management tool that is exemplary for quick and easy issue management. Although ReQtest is essentially a testing software designed to assist teams in overcoming testing obstacles, it is equally adept in facilitating project management as well as issue management. ReQtest is particularly popular among both small and big businesses for its ease of installation and configuration.

Key Features

• Contains Agile Boards that provide efficient visibility of tasks and issues alike
• Enables fast & easy searching, filtering, and categorization of bugs and issues during project management
• Generates issue reports in form of charts for a more comprehensive analysis
• Contains filters to help users quickly locate issues without wasting any time
• Allows users to visualize a preview of issues without having to click on them separately
• Single-view of all issues without the hassle of switching between boards
• Drag & Drop feature to assist easy navigation
• Capture images and videos of bugs while tracking each issue

Drawbacks

• Limited integrations.
• The pricing plan is too costly for small businesses

Pricing

• Up to 50 users – $45 per user/month
• Up to 50-100 users – $12 per user/month

DoneDone is without a doubt, the simplest issue management software available on the market. As implied by the name of the tool, Done Done aims to assist all sorts of teams, be it marketing teams or software teams in getting things done! This includes tracking of issues to be specific. It provides a central hub where teams can log issues and manage them in the seamless possible way imaginable.

Key features

• A simplified workflow that allows users to quickly familiarize with the tool and manage their way around
• Flexible views, that allow users to visualize issues logged on a daily basis, as well as on a larger scale in the form of lists
• Provides real-time in-app notifications that keep you updated on your browser, so you don’t have to constantly refer back to the app for update alerts
• Divide issues into groups of testers, priority, status, and project.
• Release Build feature allows users to coordinate issue testing and issue reporting with relevant issue assignees
• Integrations with useful apps such as Slack, GitHub, Beanstalk, and Bitbucket.

Drawbacks

• No free version available. Basic Plan too costly

Pricing

• Starter Plan – Up to 15 users. 10 GB storage. $39/ month.
• Pro Plan – Up to 50 users. 25 GB storage. $49/month

Issues in any project are as inevitable as a bump on a 100-mile road trip. With a solid issue management software, project managers and their teams alike can salvage a project’s lifeline from unforeseen hiccups that can dismantle the entire foundation of a project later down the road.

Likewise, Issues that are tracked on time can be obstructed from turning into bigger more precarious risks.

Unfortunately, our website is currently unavailable in most European countries. We are engaged on the issue and committed to looking at options that support our full range of digital offerings to the EU market. We continue to identify technical compliance solutions that will provide all readers with our award-winning journalism.

Using Deep Learning to Help Pathologists Find Tumors

By Yi Li, Wei Ping

While there are various methods to diagnose cancer, a pathology review of biopsied tissues is often considered the gold standard. However, reviewing pathology slides is not easy, even for experienced pathologists. A digitalized pathology slide at 40X magnification often contains billions of pixels, and may take up multiple gigabytes of disk space. In these massive files, pathologists sometimes have to look for micrometastases, small groups of tumor cells that are an early sign for cancer. These tumor cell groups can be smaller than 1000 pixels in diameter¹. This makes reviewing pathology slides without missing any of these tiny but clinically actionable evidences very complex and time consuming. Figure 1 illustrates this difficult task.

Various deep learning based algorithms have been proposed to aid pathologists in effectively reviewing these slides and detecting cancer metastasis. Because of the outrageously large size of the original digital slides, most of the algorithms currently being used split the slide into lots of smaller individual image patches, e.g. 256x256 pixels. A deep convolutional neural network is then trained to classify whether each small patch contains tumor cells or normal cells separately. However, sometimes it is difficult to predict whether a patch contains tumor cells without knowing its surroundings, especially around the tumor/normal boundary regions, and false positive predictions are often introduced. Figure 2 shows one example of how difficult this can be.

We have proposed a new deep learning algorithm that takes not just one individual patch but a grid of neighboring patches as input to jointly predict whether they are tumor cells or normal cells. This technique could be compared to a pathologist zooming out to see the larger field and make more confident judgements. The spatial correlations between neighboring patches are modelled through a specific type of probabilistic graphical model named conditional random fields. The whole deep learning framework can be trained end-to-end on GPU without any post processing. Figure 3 shows the architecture of the algorithm. 

By considering the spatial correlations between neighboring patches, our algorithm introduced far fewer false positives. Figure 4 shows an example of predicted tumor regions by our algorithm compared to previous algorithms that do not consider neighboring patches. We can see that our algorithm introduces very few false positives other than the ground truth tumor regions.

Figure 4. (a) the original whole slide image; (b) annotation by pathologists, where the white regions represent cancer metastasis; (c) predicted tumor regions by previous algorithms that do not consider neighboring patches; (d) predicted tumor regions by our algorithm.

On the test set of the Camelyon16 challenge, our algorithm achieved a tumor localization score (FROC) of 0.8096, which outperforms both a professional pathologist (0.7240), and the previous winner of the Camelyon16 challenge (0.8074). We are also open sourcing our algorithm on Github in the hope to advance the AI research in pathology analysis. 

This novel tumor detection algorithm has the potential to improve efficiency and accuracy in pathology slide review. It allows pathologists to focus more on tumor regions highlighted by the algorithm rather than having to search through the whole slide. Further clinical study on a larger dataset is necessary to comprehensively assess the algorithm.

For more details, please refer to our MIDL’ 2018 paper.

1.One pixel is about 0.243µm in digitalized pathology slide at 40X magnification. Micrometastases is often defined as a group of tumor cells with longest diameter larger than 200µm, which is about 823 pixels.

Well, that didn't take long to confirm. It appears I'm not the only one who's been trying to reconcile the confusing communications coming out of Nvidia lately. And as the Taiwanese supply chain has more leaks than the Trump White House, it was only a matter of time before we got the juicy details.

Tech news site SemiAccurate which covers the GPU space pretty closely, and has broken stories like AMD's acquisition of ATI Technologies and Nvidia's Bumpgate, just published an article on why Nvidia has delayed their new gaming GPUs. It seems the Hot Chips 30 agenda cancellation and Jensen's no new GPUs for 'a long time' comment have created enough of a stir to get journalists and industry insiders asking questions. While curiosity amongst all this confusion is natural, I was surprised to discover that people were starting to speculate Nvidia's delay was due to technical issues with their new GPUs. This had never been a concern of mine, and as it turns out, it's clearly not the case.

So, what the problem?

Supply...

...Too Much of It

Nvidia has overestimated pent-up gaming demand and underestimated the impact of declining mining demand.

How badly have they miscalculated?

Enough to have to agree to take back 300k GPUs from a 'top 3' Taiwan OEM. This is quite notable news as Nvidia usually exerts massive influence over its partners and can be quite ruthless with allocations of new GPUs if partners step out of line. The fact that this partner returned these GPUs says a lot about the current state of supply in the channel. The report also cites Nvidia aggressively buying GDDR5 as evidence that they now have an excess stock of lower-end GPUs that need to be made into boards as well as other insiders/sources citing an inventory buildup.

The interesting part here is that this has happened simply with mining demand tapering off and not because of some Ethereum-related flood of GPUs on the second-hand market. But now that Nvidia has inventory to work through, things should get a lot more interesting in the space. When you factor in manufacturing lead times at TSM and the current supply on hand, Nvidia really has no choice but to wait if they want to avoid expensive price protection payouts.

What does this all mean?

If you've been following my research, this isn't exactly headline-grabbing news. My Q1 analysis of Nvidia's gaming revenue and guidance clearly demonstrated management had seriously missed the boat on gaming classified mining revenue. It was inevitable that they would run into an issue like this. And it's now very clear that Jensen has a bad case of "promotional CEO syndrome" that everybody investing in this stock needs to account for going forward. I question his ability of communicating tempered, pragmatic, and grounded messages. If an approaching cryptomining-mania collapse wasn't enough to generate some caution with respect to Nvidia's gaming revenue commentary, I can't imagine what it would take to get him to acknowledge issues about the AI-competitive landscape. Google Skynet Judgement Day maybe?

As for the stock side of things, I am not exactly going to argue that you should aggressively short some shares on this news. Nvidia's drastic underperformance vs. its Nasdaq 100 peers over the past few months probably says enough. Up 12% is the new down when Apple (NASDAQ:AAPL), Google (NASDAQ:GOOG) (NASDAQ:GOOGL), Facebook (NASDAQ:FB), Intel (NASDAQ:INTC), Microsoft (NASDAQ:MSFT), Alibaba (NYSE:BABA), Amazon (NASDAQ:AMZN), and Netflix (NASDAQ:NFLX) are outperforming you in the manner they have. When every IPO doubles in a month and seemingly every other mid-tier software name is like up 100% YTD, you have to break out your bubble barometer. My short thesis here remains centered around my data center take, and I do really love the risk/reward on that going forward. But you have to ask yourself what percentage of tech stock performance these days is macro driven. I'd say right now macro is almost the whole story. If Nvidia is going to fall 20% pretty quick, what do you think the rest of these ludicrous IPOs are going to do? What about the big names in software? Intel with its now creeping AMD shares loss narrative? Even AMD after this run have their own GPU concerns???

The counter to this argument is that a rising tide like we have had lately hides bad news in plain sight and that this can turn on a dime. So, if you have spotted something material in nature on a market darling, then you should act on it. I am partial to this view, but right now, it's not exactly an easy call. Using a long pair to hedge market risk made sense two months ago, but now, you almost want to embrace risk if you are shorting. There are more efficient ways to do this than picking out individual names to short.

Disclosure:I am/we are short NVDA.

I wrote this article myself, and it expresses my own opinions. I am not receiving compensation for it (other than from Seeking Alpha). I have no business relationship with any company whose stock is mentioned in this article.

Additional disclosure: long amd, googl, bbby

(Submitted on 13 Jun 2018)

Abstract: Despite the success of neural networks (NNs), there is still a concern among many over their "black box" nature. Why do they work? Here we present a simple analytic argument that NNs are in fact essentially polynomial regression models. This view will have various implications for NNs, e.g. providing an explanation for why convergence problems arise in NNs, and it gives rough guidance on avoiding overfitting. In addition, we use this phenomenon to predict and confirm a multicollinearity property of NNs not previously reported in the literature. Most importantly, given this loose correspondence, one may choose to routinely use polynomial models instead of NNs, thus avoiding some major problems of the latter, such as having to set many tuning parameters and dealing with convergence issues. We present a number of empirical results; in each case, the accuracy of the polynomial approach matches or exceeds that of NN approaches. A many-featured, open-source software package, polyreg, is available.